One place for hosting & domains

      How To Create an Intelligent Chatbot in Python Using the spaCy NLP Library


      The author selected the COVID-19 Relief Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Interacting with software can be a daunting task in cases where there are a lot of features. In some cases, performing similar actions requires repeating steps, like navigating menus or filling forms each time an action is performed. Chatbots are virtual assistants that help users of a software system access information or perform actions without having to go through long processes. Many of these assistants are conversational, and that provides a more natural way to interact with the system.

      To create a conversational chatbot, you could use platforms like Dialogflow that help you design chatbots at a high level. Or, you can build one yourself using a library like spaCy, which is a fast and robust Python-based natural language processing (NLP) library. spaCy provides helpful features like determining the parts of speech that words belong to in a statement, finding how similar two statements are in meaning, and so on.

      In this tutorial, you will create a chatbot that not only helps users simplify their interactions with a software system, but is also intelligent enough to communicate with the user in natural language (American English in this tutorial). The chatbot will use the OpenWeather API to tell the user what the current weather is in any city of the world, but you can implement your chatbot to handle a use case with another API.

      Prerequisites

      Before you begin, you will need the following:

      This tutorial assumes you are already familiar with Python—if you would like to improve your knowledge of Python, check out our How To Code in Python 3 series. This tutorial does not require foreknowledge of natural language processing.

      Step 1 — Setting Up Your Environment

      In this step, you will install the spaCy library that will help your chatbot understand the user’s sentences.

      Having set up Python following the Prerequisites, you’ll have a virtual environment. Let’s activate that environment.

      Make sure you are in the directory where you set up your environment and then run the following command:

      • source my_env/bin/activate

      Now install spaCy:

      Finally, you will download a language model. spaCy’s language models are pre-trained NLP models that you can use to process statements to extract meaning. You’ll be working with the English language model, so you’ll download that.

      Run the following command:

      • python -m spacy download en_core_web_md

      If you run into an error like the following:

      Output

      ERROR: Failed building wheel for en-core-web-md

      You need to install wheel:

      Then download the English-language model again.

      To confirm that you have spaCy installed properly, open the Python interpreter:

      Next, import spaCy and load the English-language model:

      >>> import spacy
      >>> nlp = spacy.load("en_core_web_md")
      

      If those two statements execute without any errors, then you have spaCy installed.

      Now close the Python interpreter:

      >>> exit()
      

      You now have everything needed to begin working on the chatbot. In the next section, you’ll create a script to query the OpenWeather API for the current weather in a city.

      Step 2 — Creating the City Weather Program

      In this section, you will create a script that accepts a city name from the user, queries the OpenWeather API for the current weather in that city, and displays the response.

      First, create and open a Python file called weather_bot.py with your preferred editor:

      Next, you’ll create a function to get the current weather in a city from the OpenWeather API. This function will take the city name as a parameter and return the weather description of the city.

      Add the following code into your weather_bot.py file:

      weather_bot.py

      import requests
      
      api_key = "your_api_key"
      
      def get_weather(city_name):
          api_url = "http://api.openweathermap.org/data/2.5/weather?q={}&appid={}".format(city_name, api_key)
      
          response = requests.get(api_url)
          response_dict = response.json()
      
          weather = response_dict["weather"][0]["description"]
      
          if response.status_code == 200:
              return weather
          else:
              print('[!] HTTP {0} calling [{1}]'.format(response.status_code, api_url))
              return None
      

      First, you import the requests library, so you are able to work with and make HTTP requests. Make sure to replace your_api_key with your own API key. The next line begins the definition of the function get_weather() to retrieve the weather of the specified city.

      In this function, you construct the URL for the OpenWeather API. This URL returns the weather information (temperature, weather description, humidity, and so on) of the city and provides the result in JSON format. After that, you make a GET request to the API endpoint, store the result in a response variable, and then convert the response to a Python dictionary for easier access.

      On the next line, you extract just the weather description into a weather variable and then ensure that the status code of the API response is 200 (meaning there were no issues with the request). Finally, you return the weather description.

      If there is an issue with the request, the status code is printed out to the console, and you return None.

      To test the script, call the get_weather() function with a city of your choice (for example, London) and print the result. Add the highlighted code following your function:

      ~/weather_bot.py

      import requests
      
      def get_weather(city_name):
      
        ...
      
        return weather
      
      weather = get_weather("London")
      print(weather)
      

      Save and run the script:

      You will receive a result like the following:

      Output

      scattered clouds

      Having completed that successfully, you can now delete the last two lines from the script.

      Open it with:

      Then delete the two highlighted lines at the end of the file:

      ~/weather_bot.py

      import requests
      
      def get_weather(city_name):
      
        ...
      
        return weather
      
      weather = get_weather("London")
      print(weather)
      

      Save and close the file.

      You now have a function that returns the weather description for a particular city.

      In the next step, you’ll create a chatbot capable of figuring out whether the user wants to get the current weather in a city, and if so, the chatbot will use the get_weather() function to respond appropriately.

      Step 3 — Creating the Chatbot

      In the previous two steps, you installed spaCy and created a function for getting the weather in a specific city. Now, you will create a chatbot to interact with a user in natural language using the weather_bot.py script.

      You’ll write a chatbot() function that compares the user’s statement with a statement that represents checking the weather in a city. To make this comparison, you will use the spaCy similarity() method. This method computes the semantic similarity of two statements, that is, how similar they are in meaning. This will help you determine if the user is trying to check the weather or not.

      To begin, open the script:

      Then, import spaCy and load the English language model:

      ~/weather_bot.py

      import spacy
      import requests
      
      nlp = spacy.load("en_core_web_md")
      
      . . .
      
      

      After the get_weather() function in your file, create a chatbot() function representing the chatbot that will accept a user’s statement and return a response.

      Following your definition, add the highlighted code to create tokens for the two statements you’ll be comparing. Tokens are the different meaningful segments of a statement, like words and punctuation. This is necessary to allow spaCy to compute the semantic similarity:

      ~/weather_bot.py

      import spacy
      
      . . .
      
      def chatbot(statement):
        weather = nlp("Current weather in a city")
        statement = nlp(statement)
      

      Here the weather and statement variables contain spaCy tokens as a result of passing each corresponding string to the nlp() function.

      Save and close your file.

      Next you’ll be introducing the spaCy similarity() method to your chatbot() function. The similarity() method computes the semantic similarity of two statements as a value between 0 and 1, where a higher number means a greater similarity. You need to specify a minimum value that the similarity must have in order to be confident the user wants to check the weather.

      For example, if you check the similarity of statements 2 and 3 with statement 1 following, you get:

      1. Current weather in a city
      2. What is the weather in London? (similarity = 0.86)
      3. Peanut butter and jelly (similarity = 0.31)

      To try this for yourself, open the Python interpreter:

      Next, import spaCy and load the English-language model:

      >>> import spacy
      >>> nlp = spacy.load("en_core_web_md")
      

      Now let’s create tokens from statements 1 and 2:

      >>> statement1 = nlp("Current weather in a city")
      >>> statement2 = nlp("What is the weather in London?")
      

      Finally, let’s obtain the semantic similarity of the two statements:

      >>> print(statement1.similarity(statement2))
      

      You will receive a result like this:

      Output

      0.8557684354027663

      Setting a low minimum value (for example, 0.1) will cause the chatbot to misinterpret the user by taking statements (like statement 3) as similar to statement 1, which is incorrect. Setting a minimum value that’s too high (like 0.9) will exclude some statements that are actually similar to statement 1, such as statement 2.

      We will arbitrarily choose 0.75 for the sake of this tutorial, but you may want to test different values when working on your project.

      Let’s add this value to the script. First, open the file:

      Then add the following highlighted code to introduce the minimum value:

      ~/weather_bot.py

      import spacy
      
      . . .
      
      def chatbot(statement):
        weather = nlp("Current weather in a city")
        statement = nlp(statement)
        min_similarity = 0.75
      

      Now check if the similarity of the user’s statement to the statement about the weather is greater than or equal to the minimum similarity value you specified. Add the following highlighted if statement to check this:

      ~/weather_bot.py

      import spacy
      
      . . .
      
      def chatbot(statement):
        weather = nlp("Current weather in a city")
        statement = nlp(statement)
        min_similarity = 0.75
      
        if weather.similarity(statement) >= min_similarity:
          pass
      

      The final step is to extract the city from the user’s statement so you can pass it to the get_weather() function to retrieve the weather from the API call. Add the following highlighted for loop to implement this:

      ~/weather_bot.py

      import spacy
      
      ...
      
      def chatbot(statement):
        weather = nlp("Current weather in a city")
        statement = nlp(statement)
        min_similarity = 0.75
      
        if weather.similarity(statement) >= min_similarity:
          for ent in statement.ents:
            if ent.label_ == "GPE": # GeoPolitical Entity
              city = ent.text
              break
      

      To do this, you’re using spaCy’s named entity recognition feature. A named entity is a real-world noun that has a name, like a person, or in our case, a city. You want to extract the name of the city from the user’s statement.

      To extract the city name, you get all the named entities in the user’s statement and check which of them is a geopolitical entity (country, state, city). To do this, you loop through all the entities spaCy has extracted from the statement in the ents property, then check whether the entity label (or class) is “GPE” representing Geo-Political Entity. If it is, then you save the name of the entity (its text) in a variable called city.

      You also need to catch cases where no city was entered by adding an else block:

      ~/weather_bot.py

      import spacy
      
      ...
      
      def chatbot(statement):
        weather = nlp("Current weather in a city")
        statement = nlp(statement)
        min_similarity = 0.75
      
        if weather.similarity(statement) >= min_similarity:
          for ent in statement.ents:
            if ent.label_ == "GPE": # GeoPolitical Entity
              city = ent.text
              break
            else:
              return "You need to tell me a city to check."
      

      Now that you have the city, you can call the get_weather() function:

      ~/weather_bot.py

      import spacy
      
      ...
      
      def chatbot(statement):
        weather = nlp("Current weather in a city")
        statement = nlp(statement)
        min_similarity = 0.75
      
        if weather.similarity(statement) >= min_similarity:
          for ent in statement.ents:
            if ent.label_ == "GPE": # GeoPolitical Entity
              city = ent.text
              break
            else:
              return "You need to tell me a city to check."
      
          city_weather = get_weather(city)
          if city_weather is not None:
            return "In " + city + ", the current weather is: " + city_weather
          else:
            return "Something went wrong."
        else:
          return "Sorry I don't understand that. Please rephrase your statement."
      

      Recall that if an error is returned by the OpenWeather API, you print the error code to the terminal, and the get_weather() function returns None. In this code, you first check whether the get_weather() function returns None. If it doesn’t, then you return the weather of the city, but if it does, then you return a string saying something went wrong. The final else block is to handle the case where the user’s statement’s similarity value does not reach the threshold value. In such a case, you ask the user to rephrase their statement.

      Having completed all of that, you now have a chatbot capable of telling a user conversationally what the weather is in a city. The difference between this bot and rule-based chatbots is that the user does not have to enter the same statement every time. Instead, they can phrase their request in different ways and even make typos, but the chatbot would still be able to understand them due to spaCy’s NLP features.

      Let’s test the bot. Call the chatbot() function and pass in a statement asking what the weather is in a city, for example:

      ~/weather_bot.py

      import spacy
      
      . . .
      
      def chatbot(statement):
      
      . . .
      
      response = chatbot("Is it going to rain in Rome today?")
      print(response)
      

      Save and close the file, then run the script in your terminal:

      You will receive output similar to the following:

      Output

      In Rome, the current weather is: clear sky

      You have successfully created an intelligent chatbot capable of responding to dynamic user requests. You can try out more examples to discover the full capabilities of the bot. To do this, you can get other API endpoints from OpenWeather and other sources. Another way to extend the chatbot is to make it capable of responding to more user requests. For this, you could compare the user’s statement with more than one option and find which has the highest semantic similarity.

      Conclusion

      You have created a chatbot that is intelligent enough to respond to a user’s statement—even when the user phrases their statement in different ways. The chatbot uses the OpenWeather API to get the current weather in a city specified by the user.

      To further improve the chatbot, you can:



      Source link