One place for hosting & domains

      How To Send Web Push Notifications from Django Applications


      The author selected the Open Internet/Free Speech Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      The web is constantly evolving, and it can now achieve functionalities that were formerly only available on native mobile devices. The introduction of JavaScript service workers gave the web newfound abilities to do things like background syncing, offline caching, and sending push notifications.

      Push notifications allow users to opt-in to receive updates to mobile and web applications. They also enable users to re-engage with existing applications using customized and relevant content.

      In this tutorial, you’ll set up a Django application on Ubuntu 18.04 that sends push notifications whenever there’s an activity that requires the user to visit the application. To create these notifications, you will use the Django-Webpush package and set up and register a service worker to display notifications to the client. The working application with notifications will look like this:

      Web push final

      Prerequisites

      Before you begin this guide you’ll need the following:

      Step 1 — Installing Django-Webpush and Getting Vapid Keys

      Django-Webpush is a package that enables developers to integrate and send web push notifications in Django applications. We’ll use this package to trigger and send push notifications from our application. In this step, you will install Django-Webpush and obtain the Voluntary Application Server Identification (VAPID) keys that are necessary for identifying your server and ensuring the uniqueness of each request.

      Make sure you are in the ~/djangopush project directory that you created in the prerequisites:

      Activate your virtual environment:

      • source my_env/bin/activate

      Upgrade your version of pip to ensure it's up-to-date:

      • pip install --upgrade pip

      Install Django-Webpush:

      • pip install django-webpush

      After installing the package, add it to the list of applications in your settings.py file. First open settings.py:

      • nano ~/djangopush/djangopush/settings.py

      Add webpush to the list of INSTALLED_APPS:

      ~/djangopush/djangopush/settings.py

      ...
      
      INSTALLED_APPS = [
          ...,
          'webpush',
      ]
      ...
      

      Save the file and exit your editor.

      Run migrations on the application to apply the changes you've made to your database schema:

      The output will look like this, indicating a successful migration:

      Output

      Operations to perform: Apply all migrations: admin, auth, contenttypes, sessions, webpush Running migrations: Applying webpush.0001_initial... OK

      The next step in setting up web push notifications is getting VAPID keys. These keys identify the application server and can be used to reduce the secrecy for push subscription URLs, since they restrict subscriptions to a specific server.

      To obtain VAPID keys, navigate to the wep-push-codelab web application. Here, you'll be given automatically generated keys. Copy the private and public keys.

      Next, create a new entry in settings.py for your VAPID information. First, open the file:

      • nano ~/djangopush/djangopush/settings.py

      Next, add a new directive called WEBPUSH_SETTINGS with your VAPID public and private keys and your email below AUTH_PASSWORD_VALIDATORS:

      ~/djangopush/djangopush/settings.py

      ...
      
      AUTH_PASSWORD_VALIDATORS = [
          ...
      ]
      
      WEBPUSH_SETTINGS = {
         "VAPID_PUBLIC_KEY": "your_vapid_public_key",
         "VAPID_PRIVATE_KEY": "your_vapid_private_key",
         "VAPID_ADMIN_EMAIL": "admin@example.com"
      }
      
      # Internationalization
      # https://docs.djangoproject.com/en/2.0/topics/i18n/
      
      ...
      

      Don't forget to replace the placeholder values your_vapid_public_key, your_vapid_private_key, and admin@example.com with your own information. Your email address is how you will be notified if the push server experiences any issues.

      Next, we'll set up views that will display the application's home page and trigger push notifications to subscribed users.

      Step 2 — Setting Up Views

      In this step, we'll setup a basic home view with the HttpResponse response object for our home page, along with a send_push view. Views are functions that return response objects from web requests. The send_push view will use the Django-Webpush library to send push notifications that contain the data entered by a user on the home page.

      Navigate to the ~/djangopush/djangopush folder:

      • cd ~/djangopush/djangopush

      Running ls inside the folder will show you the project's main files:

      Output

      /__init__.py /settings.py /urls.py /wsgi.py

      The files in this folder are auto-generated by the django-admin utility that you used to create your project in the prerequisites. The settings.py file contains project-wide configurations like installed applications and the static root folder. The urls.py file contains the URL configurations for the project. This is where you will set up routes to match your created views.

      Create a new file inside the ~/djangopush/djangopush directory called views.py, which will contain the views for your project:

      • nano ~/djangopush/djangopush/views.py

      The first view we'll make is the home view, which will display the home page where users can send push notifications. Add the following code to the file:

      ~/djangopush/djangopush/views.py

      from django.http.response import HttpResponse
      from django.views.decorators.http import require_GET
      
      @require_GET
      def home(request):
          return HttpResponse('<h1>Home Page<h1>')
      

      The home view is decorated by the require_GET decorator, which restricts the view to GET requests only. A view typically returns a response for every request made to it. This view returns a simple HTML tag as a response.

      The next view we'll create is send_push, which will handle sent push notifications using the django-webpush package. It will be restricted to POST requests only and will be exempted from Cross Site Request Forgery (CSRF) protection. Doing this will allow you to test the view using Postman or any other RESTful service. In production, however, you should remove this decorator to avoid leaving your views vulnerable to CSRF.

      To create the send_push view, first add the following imports to enable JSON responses and access the send_user_notification function in the webpush library:

      ~/djangopush/djangopush/views.py

      from django.http.response import JsonResponse, HttpResponse
      from django.views.decorators.http import require_GET, require_POST
      from django.shortcuts import get_object_or_404
      from django.contrib.auth.models import User
      from django.views.decorators.csrf import csrf_exempt
      from webpush import send_user_notification
      import json
      

      Next, add the require_POST decorator, which will use the request body sent by the user to create and trigger a push notification:

      ~/djangopush/djangopush/views.py

      @require_GET
      def home(request):
          ...
      
      
      @require_POST
      @csrf_exempt
      def send_push(request):
          try:
              body = request.body
              data = json.loads(body)
      
              if 'head' not in data or 'body' not in data or 'id' not in data:
                  return JsonResponse(status=400, data={"message": "Invalid data format"})
      
              user_id = data['id']
              user = get_object_or_404(User, pk=user_id)
              payload = {'head': data['head'], 'body': data['body']}
              send_user_notification(user=user, payload=payload, ttl=1000)
      
              return JsonResponse(status=200, data={"message": "Web push successful"})
          except TypeError:
              return JsonResponse(status=500, data={"message": "An error occurred"})
      

      We are using two decorators for the send_push view: the require_POST decorator, which restricts the view to POST requests only, and the csrf_exempt decorator, which exempts the view from CSRF protection.

      This view expects POST data and does the following: it gets the body of the request and, using the json package, deserializes the JSON document to a Python object using json.loads. json.loads takes a structured JSON document and converts it to a Python object.

      The view expects the request body object to have three properties:

      • head: The title of the push notification.
      • body: The body of the notification.
      • id: The id of the request user.

      If any of the required properties are missing, the view will return a JSONResponse with a 404 "Not Found" status. If the user with the given primary key exists, the view will return the user with the matching primary key using the get_object_or_404 function from the django.shortcuts library. If the user doesn't exist, the function will return a 404 error.

      The view also makes use of the send_user_notification function from the webpush library. This function takes three parameters:

      • User: The recipient of the push notification.
      • payload: The notification information, which includes the notification head and body.
      • ttl: The maximum time in seconds that the notification should be stored if the user is offline.

      If no errors occur, the view returns a JSONResponse with a 200 "Success" status and a data object. If a KeyError occurs, the view will return a 500 "Internal Server Error" status. A KeyError occurs when the requested key of an object doesn't exist.

      In the next step, we'll create corresponding URL routes to match the views we've created.

      Step 3 — Mapping URLs to Views

      Django makes it possible to create URLs that connect to views with a Python module called a URLconf. This module maps URL path expressions to Python functions (your views). Usually, a URL configuration file is auto-generated when you create a project. In this step, you will update this file to include new routes for the views you created in the previous step, along with the URLs for the django-webpush app, which will provide endpoints to subscribe users to push notifications.

      For more information about views, please see How To Create Django Views.

      Open urls.py:

      • nano ~/djangopush/djangopush/urls.py

      The file will look like this:

      ~/djangopush/djangopush/urls.py

      
      """untitled URL Configuration
      
      The `urlpatterns` list routes URLs to views. For more information please see:
          https://docs.djangoproject.com/en/2.1/topics/http/urls/
      Examples:
      Function views
          1. Add an import:  from my_app import views
          2. Add a URL to urlpatterns:  path('', views.home, name='home')
      Class-based views
          1. Add an import:  from other_app.views import Home
          2. Add a URL to urlpatterns:  path('', Home.as_view(), name='home')
      Including another URLconf
          1. Import the include() function: from django.urls import include, path
          2. Add a URL to urlpatterns:  path('blog/', include('blog.urls'))
      """
      from django.contrib import admin
      from django.urls import path
      
      urlpatterns = [
          path('admin/', admin.site.urls),
      ]
      

      The next step is to map the views you've created to URLs. First, add the include import to ensure that all of the routes for the Django-Webpush library will be added to your project:

      ~/djangopush/djangopush/urls.py

      
      """webpushdjango URL Configuration
      ...
      """
      from django.contrib import admin
      from django.urls import path, include
      

      Next, import the views you created in the last step and update the urlpatterns list to map your views:

      ~/djangopush/djangopush/urls.py

      
      """webpushdjango URL Configuration
      ...
      """
      from django.contrib import admin
      from django.urls import path, include
      
      from .views import home, send_push
      
      urlpatterns = [
                        path('admin/', admin.site.urls),
                        path('', home),
                        path('send_push', send_push),
                        path('webpush/', include('webpush.urls')),
                    ]
      

      Here, the urlpatterns list registers the URLs for the django-webpush package and maps your views to the URLs /send_push and /home.

      Let's test the /home view to be sure that it's working as intended. Make sure you're in the root directory of the project:

      Start your server by running the following command:

      • python manage.py runserver your_server_ip:8000

      Navigate to http://your_server_ip:8000. You should see the following home page:

      Initial Home Page view

      At this point, you can kill the server with CTRL+C, and we will move on to creating templates and rendering them in our views using the render function.

      Step 4 — Creating Templates

      Django’s template engine allows you to define the user-facing layers of your application with templates, which are similar to HTML files. In this step, you will create and render a template for the home view.

      Create a folder called templates in your project's root directory:

      • mkdir ~/djangopush/templates

      If you run ls in the root folder of your project at this point, the output will look like this:

      Output

      /djangopush /templates db.sqlite3 manage.py /my_env

      Create a file called home.html in the templates folder:

      • nano ~/djangopush/templates/home.html

      Add the following code to the file to create a form where users can enter information to create push notifications:

      {% load static %}
      <!DOCTYPE html>
      <html lang="en">
      <head>
          <meta charset="UTF-8">
          <meta name="viewport" content="width=device-width, initial-scale=1.0">
          <meta http-equiv="X-UA-Compatible" content="ie=edge">
          <meta name="vapid-key" content="{{ vapid_key }}">
          {% if user.id %}
              <meta name="user_id" content="{{ user.id }}">
          {% endif %}
          <title>Web Push</title>
          <link href="https://fonts.googleapis.com/css?family=PT+Sans:400,700" rel="stylesheet">
      </head>
      
      <body>
      <div>
          <form id="send-push__form">
              <h3 class="header">Send a push notification</h3>
              <p class="error"></p>
              <input type="text" name="head" placeholder="Header: Your favorite airline 😍">
              <textarea name="body" id="" cols="30" rows="10" placeholder="Body: Your flight has been cancelled 😱😱😱"></textarea>
              <button>Send Me</button>
          </form>
      </div>
      </body>
      </html>
      

      The body of the file includes a form with two fields: an input element will hold the head/title of the notification and a textarea element will hold the notification body.

      In the head section of the file, there are two meta tags that will hold the VAPID public key and the user's id. These two variables are required to register a user and send them push notifications. The user's id is required here because you'll be sending AJAX requests to the server and the id will be used to identify the user. If the current user is a registered user, then the template will create a meta tag with their id as the content.

      The next step is to tell Django where to find your templates. To do this, you will edit settings.py and update the TEMPLATES list.

      Open the settings.py file:

      • nano ~/djangopush/djangopush/settings.py

      Add the following to the DIRS list to specify the path to the templates directory:

      ~/djangopush/djangopush/settings.py

      ...
      TEMPLATES = [
          {
              'BACKEND': 'django.template.backends.django.DjangoTemplates',
              'DIRS': [os.path.join(BASE_DIR, 'templates')],
              'APP_DIRS': True,
              'OPTIONS': {
                  'context_processors': [
                      ...
                  ],
              },
          },
      ]
      ...
      

      Next, in your views.py file, update the home view to render the home.html template. Open the file:

      • nano ~/djangpush/djangopush/views.py

      First, add some additional imports, including the settings configuration, which contains all of the project's settings from the settings.py file, and the render function from django.shortcuts:

      ~/djangopush/djangopush/views.py

      ...
      from django.shortcuts import render, get_object_or_404
      ...
      import json
      from django.conf import settings
      
      ...
      

      Next, remove the initial code you added to the home view and add the following, which specifies how the template you just created will be rendered:

      ~/djangopush/djangopush/views.py

      ...
      
      @require_GET
      def home(request):
         webpush_settings = getattr(settings, 'WEBPUSH_SETTINGS', {})
         vapid_key = webpush_settings.get('VAPID_PUBLIC_KEY')
         user = request.user
         return render(request, 'home.html', {user: user, 'vapid_key': vapid_key})
      

      The code assigns the following variables:

      • webpush_settings: This is assigned the value of the WEBPUSH_SETTINGS attribute from the settings configuration.
      • vapid_key: This gets the VAPID_PUBLIC_KEY value from the webpush_settings object to send to the client. This public key is checked against the private key to ensure that the client with the public key is permitted to receive push messages from the server.
      • user: This variable comes from the incoming request. Whenever a user makes a request to the server, the details for that user are stored in the user field.

      The render function will return an HTML file and a context object containing the current user and the server's vapid public key. It takes three parameters here: the request, the template to be rendered, and the object that contains the variables that will be used in the template.

      With our template created and the home view updated, we can move on to configuring Django to serve our static files.

      Step 5 — Serving Static Files

      Web applications include CSS, JavaScript, and other image files that Django refers to as “static files”. Django allows you to collect all of the static files from each application in your project into a single location from which they are served. This solution is called django.contrib.staticfiles. In this step, we'll update our settings to tell Django where our static files will be stored.

      Open settings.py:

      • nano ~/djangopush/djangopush/settings.py

      In settings.py, first ensure that the STATIC_URL has been defined:

      ~/djangopush/djangopush/settings.py

      ...
      STATIC_URL = '/static/'
      

      Next, add a list of directories called STATICFILES_DIRS where Django will look for static files:

      ~/djangopush/djangopush/settings.py

      ...
      STATIC_URL = '/static/'
      STATICFILES_DIRS = [
          os.path.join(BASE_DIR, "static"),
      ]
      

      You can now add the STATIC_URL to the list of paths defined in your urls.py file.

      Open the file:

      • nano ~/djangopush/djangopush/urls.py

      Add the following code, which will import the static url configuration and update the urlpatterns list. The helper function here uses the STATIC_URL and STATIC_ROOT properties we provided in the settings.py file to serve the project's static files:

      ~/djangopush/djangopush/urls.py

      
      ...
      from django.conf import settings
      from django.conf.urls.static import static
      
      urlpatterns = [
          ...
      ]  + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
      

      With our static files settings configured, we can move on to styling the application's home page.

      Step 6 — Styling the Home Page

      After setting up your application to serve static files, you can create an external stylesheet and link it to the home.html file to style the home page. All of your static files will be stored in a static directory in the root folder of your project.

      Create a static folder and a css folder within the static folder:

      • mkdir -p ~/djangopush/static/css

      Open a css file called styles.css inside the css folder:

      • nano ~/djangopush/static/css/styles.css

      Add the following styles for the home page:

      ~/djangopush/static/css/styles.css

      
      body {
          height: 100%;
          background: rgba(0, 0, 0, 0.87);
          font-family: 'PT Sans', sans-serif;
      }
      
      div {
          height: 100%;
          display: flex;
          align-items: center;
          justify-content: center;
      }
      
      form {
          display: flex;
          flex-direction: column;
          align-items: center;
          justify-content: center;
          width: 35%;
          margin: 10% auto;
      }
      
      form > h3 {
          font-size: 17px;
          font-weight: bold;
          margin: 15px 0;
          color: orangered;
          text-transform: uppercase;
      }
      
      form > .error {
          margin: 0;
          font-size: 15px;
          font-weight: normal;
          color: orange;
          opacity: 0.7;
      }
      
      form > input, form > textarea {
          border: 3px solid orangered;
          box-shadow: unset;
          padding: 13px 12px;
          margin: 12px auto;
          width: 80%;
          font-size: 13px;
          font-weight: 500;
      }
      
      form > input:focus, form > textarea:focus {
          border: 3px solid orangered;
          box-shadow: 0 2px 3px 0 rgba(0, 0, 0, 0.2);
          outline: unset;
      }
      
      form > button {
          justify-self: center;
          padding: 12px 25px;
          border-radius: 0;
          text-transform: uppercase;
          font-weight: 600;
          background: orangered;
          color: white;
          border: none;
          font-size: 14px;
          letter-spacing: -0.1px;
          cursor: pointer;
      }
      
      form > button:disabled {
          background: dimgrey;
          cursor: not-allowed;
      }
      

      With the stylesheet created, you can link it to the home.html file using static template tags. Open the home.html file:

      • nano ~/djangopush/templates/home.html

      Update the head section to include a link to the external stylesheet:

      ~/djangopush/templates/home.html

      
      {% load static %}
      <!DOCTYPE html>
      <html lang="en">
      
      <head>
          ...
          <link href="http://www.digitalocean.com/{% static"/css/styles.css' %}" rel="stylesheet">
      </head>
      <body>
          ...
      </body>
      </html>
      

      Make sure that you are in your main project directory and start your server again to inspect your work:

      • cd ~/djangopush
      • python manage.py runserver your_server_ip:8000

      When you visit http://your_server_ip:8000, it should look like this:

      Home page view
      Again, you can kill the server with CTRL+C.

      Now that you have successfully created the home.html page and styled it, you can subscribe users to push notifications whenever they visit the home page.

      Step 7 — Registering a Service Worker and Subscribing Users to Push Notifications

      Web push notifications can notify users when there are updates to applications they are subscribed to or prompt them to re-engage with applications they have used in the past. They rely on two technologies, the push API and the notifications API. Both technologies rely on the presence of a service worker.

      A push is invoked when the server provides information to the service worker and the service worker uses the notifications API to display this information.

      We'll subscribe our users to the push and then we'll send the information from the subscription to the server to register them.

      In the static directory, create a folder called js:

      • mkdir ~/djangopush/static/js

      Create a file called registerSw.js:

      • nano ~/djangopush/static/js/registerSw.js

      Add the following code, which checks if service workers are supported on the user's browser before attempting to register a service worker:

      ~/djangopush/static/js/registerSw.js

      
      const registerSw = async () => {
          if ('serviceWorker' in navigator) {
              const reg = await navigator.serviceWorker.register('sw.js');
              initialiseState(reg)
      
          } else {
              showNotAllowed("You can't send push notifications ☹️😢")
          }
      };
      

      First, the registerSw function checks if the browser supports service workers before registering them. After registration, it calls the initializeState function with the registration data. If service workers are not supported in the browser, it calls the showNotAllowed function.

      Next, add the following code below the registerSw function to check if a user is eligible to receive push notifications before attempting to subscribe them:

      ~/djangopush/static/js/registerSw.js

      
      ...
      
      const initialiseState = (reg) => {
          if (!reg.showNotification) {
              showNotAllowed('Showing notifications isn't supported ☹️😢');
              return
          }
          if (Notification.permission === 'denied') {
              showNotAllowed('You prevented us from showing notifications ☹️🤔');
              return
          }
          if (!'PushManager' in window) {
              showNotAllowed("Push isn't allowed in your browser 🤔");
              return
          }
          subscribe(reg);
      }
      
      const showNotAllowed = (message) => {
          const button = document.querySelector('form>button');
          button.innerHTML = `${message}`;
          button.setAttribute('disabled', 'true');
      };
      

      The initializeState function checks the following:

      • Whether or not the user has enabled notifications, using the value of reg.showNotification.
      • Whether or not the user has granted the application permission to display notifications.
      • Whether or not the browser supports the PushManager API.
        If any of these checks fail, the showNotAllowed function is called and the subscription is aborted.

      The showNotAllowed function displays a message on the button and disables it if a user is ineligible to receive notifications. It also displays appropriate messages if a user has restricted the application from displaying notifications or if the browser doesn't support push notifications.

      Once we ensure that a user is eligible to receive push notifications, the next step is to subscribe them using pushManager. Add the following code below the showNotAllowed function:

      ~/djangopush/static/js/registerSw.js

      
      ...
      
      function urlB64ToUint8Array(base64String) {
          const padding = '='.repeat((4 - base64String.length % 4) % 4);
          const base64 = (base64String + padding)
              .replace(/-/g, '+')
              .replace(/_/g, '/');
      
          const rawData = window.atob(base64);
          const outputArray = new Uint8Array(rawData.length);
          const outputData = outputArray.map((output, index) => rawData.charCodeAt(index));
      
          return outputData;
      }
      
      const subscribe = async (reg) => {
          const subscription = await reg.pushManager.getSubscription();
          if (subscription) {
              sendSubData(subscription);
              return;
          }
      
          const vapidMeta = document.querySelector('meta[name="vapid-key"]');
          const key = vapidMeta.content;
          const options = {
              userVisibleOnly: true,
              // if key exists, create applicationServerKey property
              ...(key && {applicationServerKey: urlB64ToUint8Array(key)})
          };
      
          const sub = await reg.pushManager.subscribe(options);
          sendSubData(sub)
      };
      

      Calling the pushManager.getSubscription function returns the data for an active subscription. When an active subscription exists, the sendSubData function is called with the subscription info passed in as a parameter.

      When no active subscription exists, the VAPID public key, which is Base64 URL-safe encoded, is converted to a Uint8Array using the urlB64ToUint8Array function. pushManager.subscribe is then called with the VAPID public key and the userVisible value as options. You can read more about the available options here.

      After successfully subscribing a user, the next step is to send the subscription data to the server. The data will be sent to the webpush/save_information endpoint provided by the django-webpush package. Add the following code below the subscribe function:

      ~/djangopush/static/js/registerSw.js

      
      ...
      
      const sendSubData = async (subscription) => {
          const browser = navigator.userAgent.match(/(firefox|msie|chrome|safari|trident)/ig)[0].toLowerCase();
          const data = {
              status_type: 'subscribe',
              subscription: subscription.toJSON(),
              browser: browser,
          };
      
          const res = await fetch('/webpush/save_information', {
              method: 'POST',
              body: JSON.stringify(data),
              headers: {
                  'content-type': 'application/json'
              },
              credentials: "include"
          });
      
          handleResponse(res);
      };
      
      const handleResponse = (res) => {
          console.log(res.status);
      };
      
      registerSw();
      

      The save_information endpoint requires information about the status of the subscription (subscribe and unsubscribe), the subscription data, and the browser. Finally, we call the registerSw() function to begin the process of subscribing the user.

      The completed file looks like this:

      ~/djangopush/static/js/registerSw.js

      
      const registerSw = async () => {
          if ('serviceWorker' in navigator) {
              const reg = await navigator.serviceWorker.register('sw.js');
              initialiseState(reg)
      
          } else {
              showNotAllowed("You can't send push notifications ☹️😢")
          }
      };
      
      const initialiseState = (reg) => {
          if (!reg.showNotification) {
              showNotAllowed('Showing notifications isn't supported ☹️😢');
              return
          }
          if (Notification.permission === 'denied') {
              showNotAllowed('You prevented us from showing notifications ☹️🤔');
              return
          }
          if (!'PushManager' in window) {
              showNotAllowed("Push isn't allowed in your browser 🤔");
              return
          }
          subscribe(reg);
      }
      
      const showNotAllowed = (message) => {
          const button = document.querySelector('form>button');
          button.innerHTML = `${message}`;
          button.setAttribute('disabled', 'true');
      };
      
      function urlB64ToUint8Array(base64String) {
          const padding = '='.repeat((4 - base64String.length % 4) % 4);
          const base64 = (base64String + padding)
              .replace(/-/g, '+')
              .replace(/_/g, '/');
      
          const rawData = window.atob(base64);
          const outputArray = new Uint8Array(rawData.length);
          const outputData = outputArray.map((output, index) => rawData.charCodeAt(index));
      
          return outputData;
      }
      
      const subscribe = async (reg) => {
          const subscription = await reg.pushManager.getSubscription();
          if (subscription) {
              sendSubData(subscription);
              return;
          }
      
          const vapidMeta = document.querySelector('meta[name="vapid-key"]');
          const key = vapidMeta.content;
          const options = {
              userVisibleOnly: true,
              // if key exists, create applicationServerKey property
              ...(key && {applicationServerKey: urlB64ToUint8Array(key)})
          };
      
          const sub = await reg.pushManager.subscribe(options);
          sendSubData(sub)
      };
      
      const sendSubData = async (subscription) => {
          const browser = navigator.userAgent.match(/(firefox|msie|chrome|safari|trident)/ig)[0].toLowerCase();
          const data = {
              status_type: 'subscribe',
              subscription: subscription.toJSON(),
              browser: browser,
          };
      
          const res = await fetch('/webpush/save_information', {
              method: 'POST',
              body: JSON.stringify(data),
              headers: {
                  'content-type': 'application/json'
              },
              credentials: "include"
          });
      
          handleResponse(res);
      };
      
      const handleResponse = (res) => {
          console.log(res.status);
      };
      
      registerSw();
      

      Next, add a script tag for the registerSw.js file in home.html. Open the file:

      • nano ~/djangopush/templates/home.html

      Add the script tag before the closing tag of the body element:

      ~/djangopush/templates/home.html

      
      {% load static %}
      <!DOCTYPE html>
      <html lang="en">
      
      <head>
         ...
      </head>
      <body>
         ...
         <script src="https://www.digitalocean.com/{% static"/js/registerSw.js' %}"></script>
      </body>
      </html>
      

      Because a service worker doesn't yet exist, if you left your application running or tried to start it again, you would see an error message. Let's fix this by creating a service worker.

      Step 8 — Creating a Service Worker

      To display a push notification, you'll need an active service worker installed on your application's home page. We'll create a service worker that listens for push events and displays the messages when ready.

      Because we want the scope of the service worker to be the entire domain, we will need to install it in the application's root. You can read more about the process in this article outlining how to register a service worker. Our approach will be to create a sw.js file in the templates folder, which we will then register as a view.

      Create the file:

      • nano ~/djangopush/templates/sw.js

      Add the following code, which tells the service worker to listen for push events:

      ~/djangopush/templates/sw.js

      
      // Register event listener for the 'push' event.
      self.addEventListener('push', function (event) {
          // Retrieve the textual payload from event.data (a PushMessageData object).
          // Other formats are supported (ArrayBuffer, Blob, JSON), check out the documentation
          // on https://developer.mozilla.org/en-US/docs/Web/API/PushMessageData.
          const eventInfo = event.data.text();
          const data = JSON.parse(eventInfo);
          const head = data.head || 'New Notification 🕺🕺';
          const body = data.body || 'This is default content. Your notification didn't have one 🙄🙄';
      
          // Keep the service worker alive until the notification is created.
          event.waitUntil(
              self.registration.showNotification(head, {
                  body: body,
                  icon: 'https://i.imgur.com/MZM3K5w.png'
              })
          );
      });
      

      The service worker listens for a push event. In the callback function, the event data is converted to text. We use default title and body strings if the event data doesn't have them. The showNotification function takes the notification title, the header of the notification to be displayed, and an options object as parameters. The options object contains several properties to configure the visual options of a notification.

      For your service worker to work for the entirety of your domain, you will need to install it in the root of the application. We'll use TemplateView to allow the service worker access to the whole domain.

      Open the urls.py file:

      • nano ~/djangopush/djangopush/urls.py

      Add a new import statement and path in the urlpatterns list to create a class-based view:

      ~/djangopush/djangopush/urls.py

      ...
      from django.views.generic import TemplateView
      
      urlpatterns = [
                        ...,
                        path('sw.js', TemplateView.as_view(template_name='sw.js', content_type='application/x-javascript'))
                    ] + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
      

      Class-based views like TemplateView allow you to create flexible, reusable views. In this case, the TemplateView.as_view method creates a path for the service worker by passing the recently created service worker as a template and application/x-javascript as the content_type of the template.

      You have now created a service worker and registered it as a route. Next, you'll set up the form on the home page to send push notifications.

      Step 9 — Sending Push Notifications

      Using the form on the home page, users should be able to send push notifications while your server is running. You can also send push notifications using any RESTful service like Postman. When the user sends push notifications from the form on the home page, the data will include a head and body, as well as the id of the receiving user. The data should be structured in the following manner:

      {
          head: "Title of the notification",
          body: "Notification body",
          id: "User's id"
      }
      

      To listen for the submit event of the form and send the data entered by the user to the server, we will create a file called site.js in the ~/djangopush/static/js directory.

      Open the file:

      • nano ~/djangopush/static/js/site.js

      First, add a submit event listener to the form that will enable you to get the values of the form inputs and the user id stored in the meta tag of your template:

      ~/djangopush/static/js/site.js

      
      const pushForm = document.getElementById('send-push__form');
      const errorMsg = document.querySelector('.error');
      
      pushForm.addEventListener('submit', async function (e) {
          e.preventDefault();
          const input = this[0];
          const textarea = this[1];
          const button = this[2];
          errorMsg.innerText = '';
      
          const head = input.value;
          const body = textarea.value;
          const meta = document.querySelector('meta[name="user_id"]');
          const id = meta ? meta.content : null;
          ...
          // TODO: make an AJAX request to send notification
      });
      

      The pushForm function gets the input, textarea, and button inside the form. It also gets the information from the meta tag, including the name attribute user_id and the user's id stored in the content attribute of the tag. With this information, it can send a POST request to the /send_push endpoint on the server.

      To send requests to the server, we'll use the native Fetch API. We're using Fetch here because it is supported by most browsers and doesn't require external libraries to function. Below the code you've added, update the pushForm function to include the code for sending AJAX requests:

      ~/djangopush/static/js/site.js

      const pushForm = document.getElementById('send-push__form');
      const errorMsg = document.querySelector('.error');
      
      pushForm.addEventListener('submit', async function (e) {
           ...
          const id = meta ? meta.content : null;
      
           if (head && body && id) {
              button.innerText = 'Sending...';
              button.disabled = true;
      
              const res = await fetch('/send_push', {
                  method: 'POST',
                  body: JSON.stringify({head, body, id}),
                  headers: {
                      'content-type': 'application/json'
                  }
              });
              if (res.status === 200) {
                  button.innerText = 'Send another 😃!';
                  button.disabled = false;
                  input.value = '';
                  textarea.value = '';
              } else {
                  errorMsg.innerText = res.message;
                  button.innerText = 'Something broke 😢..  Try again?';
                  button.disabled = false;
              }
          }
          else {
              let error;
              if (!head || !body){
                  error = 'Please ensure you complete the form 🙏🏾'
              }
              else if (!id){
                  error = "Are you sure you're logged in? 🤔. Make sure! 👍🏼"
              }
              errorMsg.innerText = error;
          }
      });
      

      If the three required parameters head, body, and id are present, we send the request and disable the submit button temporarily.

      The completed file looks like this:

      ~/djangopush/static/js/site.js

      const pushForm = document.getElementById('send-push__form');
      const errorMsg = document.querySelector('.error');
      
      pushForm.addEventListener('submit', async function (e) {
          e.preventDefault();
          const input = this[0];
          const textarea = this[1];
          const button = this[2];
          errorMsg.innerText = '';
      
          const head = input.value;
          const body = textarea.value;
          const meta = document.querySelector('meta[name="user_id"]');
          const id = meta ? meta.content : null;
      
          if (head && body && id) {
              button.innerText = 'Sending...';
              button.disabled = true;
      
              const res = await fetch('/send_push', {
                  method: 'POST',
                  body: JSON.stringify({head, body, id}),
                  headers: {
                      'content-type': 'application/json'
                  }
              });
              if (res.status === 200) {
                  button.innerText = 'Send another 😃!';
                  button.disabled = false;
                  input.value = '';
                  textarea.value = '';
              } else {
                  errorMsg.innerText = res.message;
                  button.innerText = 'Something broke 😢..  Try again?';
                  button.disabled = false;
              }
          }
          else {
              let error;
              if (!head || !body){
                  error = 'Please ensure you complete the form 🙏🏾'
              }
              else if (!id){
                  error = "Are you sure you're logged in? 🤔. Make sure! 👍🏼"
              }
              errorMsg.innerText = error;
          }    
      });
      

      Finally, add the site.js file to home.html:

      • nano ~/djangopush/templates/home.html

      Add the script tag:

      ~/djangopush/templates/home.html

      
      {% load static %}
      <!DOCTYPE html>
      <html lang="en">
      
      <head>
         ...
      </head>
      <body>
         ...
         <script src="https://www.digitalocean.com/{% static"/js/site.js' %}"></script>
      </body>
      </html>
      

      At this point, if you left your application running or tried to start it again, you would see an error, since service workers can only function in secure domains or on localhost. In the next step we'll use ngrok to create a secure tunnel to our web server.

      Step 10 — Creating a Secure Tunnel to Test the Application

      Service workers require secure connections to function on any site except localhost since they can allow connections to be hijacked and responses to be filtered and fabricated. For this reason, we'll create a secure tunnel for our server with ngrok.

      Open a second terminal window and ensure you're in your home directory:

      If you started with a clean 18.04 server in the prerequisites, then you will need to install unzip:

      • sudo apt update && sudo apt install unzip

      Download ngrok:

      • wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
      • unzip ngrok-stable-linux-amd64.zip

      Move ngrok to /usr/local/bin, so that you will have access to the ngrok command from the terminal:

      • sudo mv ngrok /usr/local/bin

      In your first terminal window, make sure that you are in your project directory and start your server:

      • cd ~/djangopush
      • python manage.py runserver your_server_ip:8000

      You will need to do this before creating a secure tunnel for your application.

      In your second terminal window, navigate to your project folder, and activate your virtual environment:

      • cd ~/djangopush
      • source my_env/bin/activate

      Create the secure tunnel to your application:

      • ngrok http your_server_ip:8000

      You will see the following output, which includes information about your secure ngrok URL:

      Output

      ngrok by @inconshreveable (Ctrl+C to quit) Session Status online Session Expires 7 hours, 59 minutes Version 2.2.8 Region United States (us) Web Interface http://127.0.0.1:4040 Forwarding http://ngrok_secure_url -> 203.0.113.0:8000 Forwarding https://ngrok_secure_url -> 203.0.113.0:8000 Connections ttl opn rt1 rt5 p50 p90 0 0 0.00 0.00 0.00 0.00

      Copy the ngrok_secure_url from the console output. You will need to add it to the list of ALLOWED_HOSTS in your settings.py file.

      Open another terminal window, navigate to your project folder, and activate your virtual environment:

      • cd ~/djangopush
      • source my_env/bin/activate

      Open the settings.py file:

      • nano ~/djangopush/djangopush/settings.py

      Update the list of ALLOWED_HOSTS with the ngrok secure tunnel:

      ~/djangopush/djangopush/settings.py

      ...
      
      ALLOWED_HOSTS = ['your_server_ip', 'ngrok_secure_url']
      ...
      
      

      Navigate to the secure admin page to log in: https://ngrok_secure_url/admin/. You will see a screen that looks like this:

      ngrok admin login

      Enter your Django admin user information on this screen. This should be the same information you entered when you logged into the admin interface in the prerequisite steps. You are now ready to send push notifications.

      Visit https://ngrok_secure_url in your browser. You will see a prompt asking for permission to display notifications. Click the Allow button to let your browser display push notifications:

      push notifications request

      Submitting a filled form will display a notification similar to this:

      screenshot of notification

      Note: Be sure that your server is running before attempting to send notifications.

      If you received notifications then your application is working as expected.

      You have created a web application that triggers push notifications on the server and, with the help of service workers, receives and displays notifications. You also went through the steps of obtaining the VAPID keys that are required to send push notifications from an application server.

      Conclusion

      In this tutorial, you've learned how to subscribe users to push notifications, install service workers, and display push notifications using the notifications API.

      You can go even further by configuring the notifications to open specific areas of your application when clicked. The source code for this tutorial can be found here.



      Source link

      How To Build a Modern Web Application to Manage Customer Information with Django and React on Ubuntu 18.04


      The author selected Open Sourcing Mental Illness Ltd to receive a donation as part of the Write for DOnations program.

      Introduction

      People use different types of devices to connect to the internet and browse the Web. Because of this, applications need to be accessible from a variety of locations. For traditional websites, having a responsive UI is usually enough, but more complex applications often require the use of other techniques and architectures. These include having separate REST back-end and front-end applications that can be implemented as client-side web applications, Progressive Web Apps (PWAs), or native mobile apps.

      Some tools that you can use when building more complex applications include:

      • React, a JavaScript framework that allows developers to build web and native frontends for their REST API backends.
      • Django, a free and open-source Python web framework that follows the model view controller (MVC) software architectural pattern.
      • Django REST framework, a powerful and flexible toolkit for building REST APIs in Django.

      In this tutorial, you will build a modern web application with a separate REST API backend and frontend using React, Django, and the Django REST Framework. By using React with Django, you’ll be able to benefit from the latest advancements in JavaScript and front-end development. Instead of building a Django application that uses a built-in template engine, you will use React as a UI library, taking advantage of its virtual Document Object Model (DOM), declarative approach, and components that quickly render changes in data.

      The web application you will build stores records about customers in a database, and you can use it as a starting point for a CRM application. When you are finished you’ll be able to create, read, update, and delete records using a React interface styled with Bootstrap 4.

      Prerequisites

      To complete this tutorial, you will need:

      Step 1 — Creating a Python Virtual Environment and Installing Dependencies

      In this step, we’ll create a virtual environment and install the required dependencies for our application, including Django, the Django REST framework, and django-cors-headers.

      Our application will use two different development servers for Django and React. They will run on different ports and will function as two separate domains. Because of this, we need to enable cross-origin resource sharing (CORS) to send HTTP requests from React to Django without being blocked by the browser.

      Navigate to your home directory and create a virtual environment using the venv Python 3 module:

      • cd ~
      • python3 -m venv ./env

      Activate the created virtual environment using source:

      Next, install the project's dependencies with pip. These will include:

      • Django: The web framework for the project.
      • Django REST framework: A third-party application that builds REST APIs with Django.
      • django-cors-headers: A package that enables CORS.

      Install the Django framework:

      • pip install django djangorestframework django-cors-headers

      With the project dependencies installed, you can create the Django project and the React frontend.

      Step 2 — Creating the Django Project

      In this step, we'll generate the Django project using the following commands and utilities:

      • django-admin startproject project-name: django-admin is a command-line utility used to accomplish tasks with Django. The startproject command creates a new Django project.

      • python manage.py startapp myapp: manage.py is a utility script, automatically added to each Django project, that performs a number of administrative tasks: creating new applications, migrating the database, and serving the Django project locally. Its startapp command creates a Django application inside the Django project. In Django, the term application describes a Python package that provides some set of features in a project.

      To begin, create the Django project with django-admin startproject. We will call our project djangoreactproject:

      • django-admin startproject djangoreactproject

      Before moving on, let's look at the directory structure of our Django project using the tree command.

      Tip: tree is a useful command for viewing file and directory structures from the command line. You can install it with the following command:

      • sudo apt-get install tree

      To use it, cd into the directory you want and type tree or provide the path to the starting point with tree /home/sammy/sammys-project.

      Navigate to the djangoreactproject folder within your project root and run the tree command:

      • cd ~/djangoreactproject
      • tree

      You will see the following output:

      Output

      ├── djangoreactproject │ ├── __init__.py │ ├── settings.py │ ├── urls.py │ └── wsgi.py └── manage.py

      The ~/djangoreactproject folder is the root of the project. Within this folder, there are several files that will be important to your work:

      • manage.py: The utility script that does a number of administrative tasks.
      • settings.py: The main configuration file for the Django project where you can modify the project's settings. These settings include variables such as INSTALLED_APPS, a list of strings designating the enabled applications for your project. The Django documentation has more information about available settings.
      • urls.py: This file contains a list of URL patterns and related views. Each pattern maps a connection between a URL and the function that should be called for that URL. For more on URLs and views, please refer to our tutorial on How To Create Django Views.

      Our first step in working with the project will be to configure the packages we installed in the previous step, including the Django REST framework and the Django CORS package, by adding them to settings.py. Open the file with nano or your favorite editor:

      • nano ~/djangoreactproject/djangoreactproject/settings.py

      Navigate to the INSTALLED_APPS setting and add the rest_framework and corsheaders applications to the bottom of the list:

      ~/djangoreactproject/djangoreactproject/settings.py

      ...
      INSTALLED_APPS = [
          'django.contrib.admin',
          'django.contrib.auth',
          'django.contrib.contenttypes',
          'django.contrib.sessions',
          'django.contrib.messages',
          'django.contrib.staticfiles',
          'rest_framework',
          'corsheaders'
      ]
      

      Next, add the corsheaders.middleware.CorsMiddleware middleware from the previously installed CORS package to the MIDDLEWARE setting. This setting is a list of middlewares, a Python class that contains code processed each time your web application handles a request or response:

      ~/djangoreactproject/djangoreactproject/settings.py

      ...
      
      MIDDLEWARE = [
      ...
      'django.contrib.messages.middleware.MessageMiddleware',
      'django.middleware.clickjacking.XFrameOptionsMiddleware',
      'corsheaders.middleware.CorsMiddleware'
      ]
      

      Next, you can enable CORS. The CORS_ORIGIN_ALLOW_ALL setting specifies whether or not you want to allow CORS for all domains, and CORS_ORIGIN_WHITELIST is a Python tuple that contains allowed URLs. In our case, because the React development server will be running at http://localhost:3000, we will add new CORS_ORIGIN_ALLOW_ALL = False and CORS_ORIGIN_WHITELIST('localhost:3000',) settings to our settings.py file. Add these settings anywhere in the file:

      ~/djangoreactproject/djangoreactproject/settings.py

      
      ...
      CORS_ORIGIN_ALLOW_ALL = False
      
      CORS_ORIGIN_WHITELIST = (
             'localhost:3000',
      )
      ...
      

      You can find more configuration options in the django-cors-headers docs.

      Save the file and exit the editor when you are finished.

      Still in the ~/djangoreactproject directory, make a new Django application called customers:

      • python manage.py startapp customers

      This will contain the models and views for managing customers. Models define the fields and behaviors of our application data, while views enable our application to properly handle web requests and return the required responses.

      Next, add this application to the list of installed applications in your project's settings.py file so Django will recognize it as part of the project. Open settings.py again:

      • nano ~/djangoreactproject/djangoreactproject/settings.py

      Add the customers application:

      ~/djangoreactproject/djangoreactproject/settings.py

      ...
      INSTALLED_APPS = [
          ...
          'rest_framework',
          'corsheaders',
          'customers'
      ]
      ...
      

      Next, migrate the database and start the local development server. Migrations are Django’s way of propagating the changes you make to your models into your database schema. These changes can include things like adding a field or deleting a model, for example. For more on models and migrations, see How To Create Django Models.

      Migrate the database:

      Start the local development server:

      • python manage.py runserver

      You will see output similar to the following:

      Output

      Performing system checks... System check identified no issues (0 silenced). October 22, 2018 - 15:14:50 Django version 2.1.2, using settings 'djangoreactproject.settings' Starting development server at http://127.0.0.1:8000/ Quit the server with CONTROL-C.

      Your web application will be running from http://127.0.0.1:8000. If you navigate to this address in your web browser you should see the following page:

      Django demo page

      At this point, leave the application running and open a new terminal to continue developing the project.

      Step 3 — Creating the React Frontend

      In this section, we're going to create the front-end application of our project using React.

      React has an official utility that allows you to quickly generate React projects without having to configure Webpack directly. Webpack is a module bundler used to bundle web assets such as JavaScript code, CSS, and images. Typically, before you can use Webpack you need to set various configuration options, but thanks to the create-react-app utility you don't have to deal with Webpack directly until you decide you need more control. To run create-react-app you can use npx, a tool that executes npm package binaries.

      In your second terminal, make sure you are in your project directory:

      Create a React project called frontend using create-react-app and npx:

      • npx create-react-app frontend

      Next, navigate inside your React application and start the development server:

      • cd ~/djangoreactproject/frontend
      • npm start

      You application will be running from http://localhost:3000/:

      React demo page

      Leave the React development server running and open another terminal window to proceed.

      To see the directory structure of the entire project at this point, navigate to the root folder and run tree again:

      • cd ~/djangoreactproject
      • tree

      You'll see a structure like this:

      Output

      ├── customers │ ├── admin.py │ ├── apps.py │ ├── __init__.py │ ├── migrations │ │ └── __init__.py │ ├── models.py │ ├── tests.py │ └── views.py ├── djangoreactproject │ ├── __init__.py │ ├── __pycache__ │ ├── settings.py │ ├── urls.py │ └── wsgi.py ├── frontend │ ├── package.json │ ├── public │ │ ├── favicon.ico │ │ ├── index.html │ │ └── manifest.json │ ├── README.md │ ├── src │ │ ├── App.css │ │ ├── App.js │ │ ├── App.test.js │ │ ├── index.css │ │ ├── index.js │ │ ├── logo.svg │ │ └── registerServiceWorker.js │ └── yarn.lock └── manage.py

      Our application will use Bootstrap 4 to style the React interface, so we will include it in the frontend/src/App.css file, which manages our CSS settings. Open the file:

      • nano ~/djangoreactproject/frontend/src/App.css

      Add the following import to the beginning of the file. You can delete the file's existing content, though that's not required:

      ~/djangoreactproject/frontend/src/App.css

      @import  'https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css';
      

      Here, @import is a CSS instruction that's used to import style rules from other style sheets.

      Now that we have created both the back-end and front-end applications, let's create the Customer model and some demo data.

      Step 4 — Creating the Customer Model and Initial Data

      After creating the Django application and the React frontend, our next step will be to create the Customer model, which represents the database table that will hold information about customers. You don't need any SQL since the Django Object Relational Mapper (ORM) will handle database operations by mapping Python classes and variables to SQL tables and columns. In this way the Django ORM abstracts SQL interactions with the database through a Python interface.

      Activate your virtual environment again:

      • cd ~
      • source env/bin/activate

      Move to the customers directory, and open models.py, a Python file that holds the models of your application:

      • cd ~/djangoreactproject/customers/
      • nano models.py

      The file will contain the following content:

      ~/djangoreactproject/customers/models.py

      from django.db import models
      # Create your models here.
      

      The Customer model's API is already imported in the file thanks to the from django.db import models import statement. You will now add the Customer class, which extends models.Model. Each model in Django is a Python class that extends django.db.models.Model.

      The Customer model will have these database fields:

      • first_name — The first name of the customer.
      • last_name — The last name of the customer.
      • email — The email address of the customer.
      • phone — The phone number of the customer.
      • address — The address of the customer.
      • description — The description of the customer.
      • createdAt — The date when the customer is added.

      We will also add the __str__() function, which defines how the model will be displayed. In our case, it will be with the customer's first name. For more on constructing classes and defining objects, please see How To Construct Classes and Define Objects in Python 3.

      Add the following code to the file:

      ~/djangoreactproject/customers/models.py

      from django.db import models
      
      class Customer(models.Model):
          first_name = models.CharField("First name", max_length=255)
          last_name = models.CharField("Last name", max_length=255)
          email = models.EmailField()
          phone = models.CharField(max_length=20)
          address =  models.TextField(blank=True, null=True)
          description = models.TextField(blank=True, null=True)
          createdAt = models.DateTimeField("Created At", auto_now_add=True)
      
          def __str__(self):
              return self.first_name
      

      Next, migrate the database to create the database tables. The makemigrations command creates the migration files where model changes will be added, and migrate applies the changes in the migrations files to the database.

      Navigate back to the project's root folder:

      Run the following to create the migration files:

      • python manage.py makemigrations

      You will get output that looks like this:

      Output

      customers/migrations/0001_initial.py - Create model Customer

      Apply these changes to the database:

      You will see output indicating a successful migration:

      Output

      Operations to perform: Apply all migrations: admin, auth, contenttypes, customers, sessions Running migrations: Applying customers.0001_initial... OK

      Next, you will use a data migration file to create initial customer data. A data migration file is a migration that adds or alters data in the database. Create an empty data migration file for the customers application:

      • python manage.py makemigrations --empty --name customers customers

      You will see the following confirmation with the name of your migration file:

      Output

      Migrations for 'customers': customers/migrations/0002_customers.py

      Note that the name of your migration file is 0002_customers.py.

      Next, navigate inside the migrations folder of the customers application:

      • cd ~/djangoreactproject/customers/migrations

      Open the created migration file:

      This is the initial content of the file:

      ~/djangoreactproject/customers/migrations/0002_customers.py

      from django.db import migrations
      
      class Migration(migrations.Migration):
          dependencies = [
              ('customers', '0001_initial'),
          ]
          operations = [
          ]        
      

      The import statement imports the migrations API, a Django API for creating migrations, from django.db, a built-in package that contains classes for working with databases.

      The Migration class is a Python class that describes the operations that are executed when migrating databases. This class extends migrations.Migration and has two lists:

      • dependencies: Contains the dependent migrations.
      • operations: Contains the operations that will be executed when we apply the migration.

      Next, add a method to create demo customer data. Add the following method before the definition of the Migration class:

      ~/djangoreactproject/customers/migrations/0002_customers.py

      ...
      def create_data(apps, schema_editor):
          Customer = apps.get_model('customers', 'Customer')
          Customer(first_name="Customer 001", last_name="Customer 001", email="customer001@email.com", phone="00000000", address="Customer 000 Address", description= "Customer 001 description").save()
      
      ...
      

      In this method, we are grabbing the Customer class of our customers app and creating a demo customer to insert into the database.

      To get the Customer class, which will enable the creation of new customers, we use the get_model() method of the apps object. The apps object represents the registry of installed applications and their database models.

      The apps object will be passed from the RunPython() method when we use it to run create_data(). Add the migrations.RunPython() method to the empty operations list:

      ~/djangoreactproject/customers/migrations/0002_customers.py

      
      ...
          operations = [
              migrations.RunPython(create_data),
          ]  
      

      RunPython() is part of the Migrations API that allows you to run custom Python code in a migration. Our operations list specifies that this method will be executed when we apply the migration.

      This is the complete file:

      ~/djangoreactproject/customers/migrations/0002_customers.py

      from django.db import migrations
      
      def create_data(apps, schema_editor):
          Customer = apps.get_model('customers', 'Customer')
          Customer(first_name="Customer 001", last_name="Customer 001", email="customer001@email.com", phone="00000000", address="Customer 000 Address", description= "Customer 001 description").save()
      
      class Migration(migrations.Migration):
          dependencies = [
              ('customers', '0001_initial'),
          ]
          operations = [
              migrations.RunPython(create_data),
          ]        
      

      For more information on data migrations, see the documentation on data migrations in Django

      To migrate your database, first navigate back to the root folder of your project:

      Migrate your database to create the demo data:

      You will see output that confirms the migration:

      Output

      Operations to perform: Apply all migrations: admin, auth, contenttypes, customers, sessions Running migrations: Applying customers.0002_customers... OK

      For more details on this process, refer back to How To Create Django Models.

      With the Customer model and demo data created, we can move on to building the REST API.

      Step 5 — Creating the REST API

      In this step we'll create the REST API using the Django REST Framework. We'll create several different API views. An API view is a function that handles an API request or call, while an API endpoint is a unique URL that represents a touchpoint with the REST system. For example, when the user sends a GET request to an API endpoint, Django calls the corresponding function or API view to handle the request and return any possible results.

      We'll also make use of serializers. A serializer in the Django REST Framework allows complex model instances and QuerySets to be converted into JSON format for API consumption. The serializer class can also work in the other direction, providing mechanisms for parsing and deserializing data into Django models and QuerySets.

      Our API endpoints will include:

      • api/customers: This endpoint is used to create customers and returns paginated sets of customers.
      • api/customers/<pk>: This endpoint is used to get, update, and delete single customers by primary key or id.

      We'll also create URLs in the project's urls.py file for the corresponding endpoints (i.e api/customers and api/customers/<pk>).

      Let's start by creating the serializer class for our Customer model.

      Adding the Serializer Class

      Creating a serializer class for our Customer model is necessary for transforming customer instances and QuerySets to and from JSON. To create the serializer class, first make a serializers.py file inside the customers application:

      • cd ~/djangoreactproject/customers/
      • nano serializers.py

      Add the following code to import the serializers API and Customer model:

      ~/djangoreactproject/customers/serializers.py

      from rest_framework import serializers
      from .models import Customer
      

      Next, create a serializer class that extends serializers.ModelSerializer and specifies the fields that will be serialized:

      ~/djangoreactproject/customers/serializers.py

      
      ...
      class CustomerSerializer(serializers.ModelSerializer):
      
          class Meta:
              model = Customer 
              fields = ('pk','first_name', 'last_name', 'email', 'phone','address','description')
      

      The Meta class specifies the model and fields to serialize: pk,first_name, last_name, email, phone, address,description.

      This is the full content of the file:

      ~/djangoreactproject/customers/serializers.py

      from rest_framework import serializers
      from .models import Customer
      
      class CustomerSerializer(serializers.ModelSerializer):
      
          class Meta:
              model = Customer 
              fields = ('pk','first_name', 'last_name', 'email', 'phone','address','description')
      

      Now that we've created our serializer class, we can add the API views.

      Adding the API Views

      In this section, we'll create the API views for our application that will be called by Django when the user visits the endpoint corresponding to the view function.

      Open ~/djangoreactproject/customers/views.py:

      • nano ~/djangoreactproject/customers/views.py

      Delete what's there and add the following imports:

      ~/djangoreactproject/customers/views.py

      from rest_framework.response import Response
      from rest_framework.decorators import api_view
      from rest_framework import status
      
      from django.core.paginator import Paginator, EmptyPage, PageNotAnInteger
      from .models import Customer 
      from .serializers import *
      

      We are importing the serializer we created, along with the Customer model and the Django and Django REST Framework APIs.

      Next, add the view for processing POST and GET HTTP requests:

      ~/djangoreactproject/customers/views.py

      ...
      
      @api_view(['GET', 'POST'])
      def customers_list(request):
          """
       List  customers, or create a new customer.
       """
          if request.method == 'GET':
              data = []
              nextPage = 1
              previousPage = 1
              customers = Customer.objects.all()
              page = request.GET.get('page', 1)
              paginator = Paginator(customers, 10)
              try:
                  data = paginator.page(page)
              except PageNotAnInteger:
                  data = paginator.page(1)
              except EmptyPage:
                  data = paginator.page(paginator.num_pages)
      
              serializer = CustomerSerializer(data,context={'request': request} ,many=True)
              if data.has_next():
                  nextPage = data.next_page_number()
              if data.has_previous():
                  previousPage = data.previous_page_number()
      
              return Response({'data': serializer.data , 'count': paginator.count, 'numpages' : paginator.num_pages, 'nextlink': '/api/customers/?page=' + str(nextPage), 'prevlink': '/api/customers/?page=' + str(previousPage)})
      
          elif request.method == 'POST':
              serializer = CustomerSerializer(data=request.data)
              if serializer.is_valid():
                  serializer.save()
                  return Response(serializer.data, status=status.HTTP_201_CREATED)
              return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)
      

      First we use the @api_view(['GET', 'POST']) decorator to create an API view that can accept GET and POST requests. A decorator is a function that takes another function and dynamically extends it.

      In the method body we use the request.method variable to check the current HTTP method and execute the corresponding logic depending on the request type:

      • If it's a GET request, the method paginates the data using Django Paginator, and returns the first page of data after serialization, the count of available customers, the number of available pages, and the links to the previous and next pages. Paginator is a built-in Django class that paginates a list of data into pages and provides methods to access the items for each page.
      • If it's a POST request, the method serializes the received customer data and then calls the save() method of the serializer object. It then returns a Response object, an instance of HttpResponse, with a 201 status code. Each view you create is responsible for returing an HttpResponse object. The save() method saves the serialized data in the database.

      For more about HttpResponse and views, see this discussion of creating view functions.

      Now add the API view that will be responsible for processing the GET, PUT, and DELETE requests for getting, updating, and deleting customers by pk (primary key):

      ~/djangoreactproject/customers/views.py

      
      ...
      @api_view(['GET', 'PUT', 'DELETE'])
      def customers_detail(request, pk):
       """
       Retrieve, update or delete a customer by id/pk.
       """
          try:
              customer = Customer.objects.get(pk=pk)
          except Customer.DoesNotExist:
              return Response(status=status.HTTP_404_NOT_FOUND)
      
          if request.method == 'GET':
              serializer = CustomerSerializer(customer,context={'request': request})
              return Response(serializer.data)
      
          elif request.method == 'PUT':
              serializer = CustomerSerializer(customer, data=request.data,context={'request': request})
              if serializer.is_valid():
                  serializer.save()
                  return Response(serializer.data)
              return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)
      
          elif request.method == 'DELETE':
              customer.delete()
              return Response(status=status.HTTP_204_NO_CONTENT)
      

      The method is decorated with @api_view(['GET', 'PUT', 'DELETE']) to denote that it's an API view that can accept GET, PUT, and DELETE requests.

      The check in the request.method field verifies the request method, and depending on its value calls the right logic:

      • If it's a GET request, customer data is serialized and sent using a Response object.
      • If it's a PUT request, the method creates a serializer for new customer data. Next, it calls the save() method of the created serializer object. Finally, it sends a Response object with the updated customer.
      • If it's a DELETE request, the method calls the delete() method of the customer object to delete it, then returns a Response object with no data.

      The completed file looks like this:

      ~/djangoreactproject/customers/views.py

      from rest_framework.response import Response
      from rest_framework.decorators import api_view
      from rest_framework import status
      
      from django.core.paginator import Paginator, EmptyPage, PageNotAnInteger
      from .models import Customer 
      from .serializers import *
      
      
      @api_view(['GET', 'POST'])
      def customers_list(request):
          """
       List  customers, or create a new customer.
       """
          if request.method == 'GET':
              data = []
              nextPage = 1
              previousPage = 1
              customers = Customer.objects.all()
              page = request.GET.get('page', 1)
              paginator = Paginator(customers, 5)
              try:
                  data = paginator.page(page)
              except PageNotAnInteger:
                  data = paginator.page(1)
              except EmptyPage:
                  data = paginator.page(paginator.num_pages)
      
              serializer = CustomerSerializer(data,context={'request': request} ,many=True)
              if data.has_next():
                  nextPage = data.next_page_number()
              if data.has_previous():
                  previousPage = data.previous_page_number()
      
              return Response({'data': serializer.data , 'count': paginator.count, 'numpages' : paginator.num_pages, 'nextlink': '/api/customers/?page=' + str(nextPage), 'prevlink': '/api/customers/?page=' + str(previousPage)})
      
          elif request.method == 'POST':
              serializer = CustomerSerializer(data=request.data)
              if serializer.is_valid():
                  serializer.save()
                  return Response(serializer.data, status=status.HTTP_201_CREATED)
              return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)
      
      @api_view(['GET', 'PUT', 'DELETE'])
      def customers_detail(request, pk):
          """
       Retrieve, update or delete a customer by id/pk.
       """
          try:
              customer = Customer.objects.get(pk=pk)
          except Customer.DoesNotExist:
              return Response(status=status.HTTP_404_NOT_FOUND)
      
          if request.method == 'GET':
              serializer = CustomerSerializer(customer,context={'request': request})
              return Response(serializer.data)
      
          elif request.method == 'PUT':
              serializer = CustomerSerializer(customer, data=request.data,context={'request': request})
              if serializer.is_valid():
                  serializer.save()
                  return Response(serializer.data)
              return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)
      
          elif request.method == 'DELETE':
              customer.delete()
              return Response(status=status.HTTP_204_NO_CONTENT)
      

      We can now move on to creating our endpoints.

      Adding API Endpoints

      We will now create the API endpoints: api/customers/, for querying and creating customers, and api/customers/<pk>, for getting, updating, or deleting single customers by their pk.

      Open ~/djangoreactproject/djangoreactproject/urls.py:

      • nano ~/djangoreactproject/djangoreactproject/urls.py

      Leave what's there, but add the import to the customers views at the top of the file:

      ~/djangoreactproject/djangoreactproject/urls.py

      from django.contrib import admin
      from django.urls import path
      from customers import views
      from django.conf.urls import url
      

      Next, add the api/customers/ and api/customers/<pk> URLs to the urlpatterns list that contains the application's URLs:

      ~/djangoreactproject/djangoreactproject/urls.py

      ...
      
      urlpatterns = [
          path('admin/', admin.site.urls),
          url(r'^api/customers/$', views.customers_list),
          url(r'^api/customers/(?P<pk>[0-9]+)$', views.customers_detail),
      ]
      

      With our REST endpoints created, let's see how we can consume them.

      Step 6 — Consuming the REST API with Axios

      In this step, we'll install Axios, the HTTP client we'll use to make API calls. We'll also create a class to consume the API endpoints we've created.

      First, deactivate your virtual environment:

      Next, navigate to your frontend folder:

      • cd ~/djangoreactproject/frontend

      Install axios from npm using:

      The --save option adds the axios dependency to your application's package.json file.

      Next, create a JavaScript file called CustomersService.js, which will contain the code to call the REST APIs. We'll make this inside the src folder, where the application code for our project will live:

      • cd src
      • nano CustomersService.js

      Add the following code, which contains methods to connect to the Django REST API:

      ~/djangoreactproject/frontend/src/CustomersService.js

      import axios from 'axios';
      const API_URL = 'http://localhost:8000';
      
      export default class CustomersService{
      
          constructor(){}
      
      
          getCustomers() {
              const url = `${API_URL}/api/customers/`;
              return axios.get(url).then(response => response.data);
          }  
          getCustomersByURL(link){
              const url = `${API_URL}${link}`;
              return axios.get(url).then(response => response.data);
          }
          getCustomer(pk) {
              const url = `${API_URL}/api/customers/${pk}`;
              return axios.get(url).then(response => response.data);
          }
          deleteCustomer(customer){
              const url = `${API_URL}/api/customers/${customer.pk}`;
              return axios.delete(url);
          }
          createCustomer(customer){
              const url = `${API_URL}/api/customers/`;
              return axios.post(url,customer);
          }
          updateCustomer(customer){
              const url = `${API_URL}/api/customers/${customer.pk}`;
              return axios.put(url,customer);
          }
      }
      

      The CustomersService class will call the following Axios methods:

      • getCustomers(): Gets first page of customers.
      • getCustomersByURL(): Gets customers by URL. This makes it possible to get the next pages of customers by passing links such as /api/customers/?page=2.
      • getCustomer(): Gets a customer by primary key.
      • createCustomer(): Creates a customer.
      • updateCustomer(): Updates a customer.
      • deleteCustomer(): Deletes a customer.

      We can now display the data from our API in our React UI interface by creating a CustomersList component.

      Step 7 — Displaying Data from the API in the React Application

      In this step, we'll create the CustomersList React component. A React component represents a part of the UI; it also lets you split the UI into independent, reusable pieces.

      Begin by creating CustomersList.js in frontend/src:

      • nano ~/djangoreactproject/frontend/src/CustomersList.js

      Start by importing React and Component to create a React component:

      ~/djangoreactproject/frontend/src/CustomersList.js

      import  React, { Component } from  'react';
      

      Next, import and instantiate the CustomersService module you created in the previous step, which provides methods that interface with the REST API backend:

      ~/djangoreactproject/frontend/src/CustomersList.js

      
      ...
      import  CustomersService  from  './CustomersService';
      
      const  customersService  =  new  CustomersService();
      

      Next, create a CustomersList component that extends Component to call the REST API. A React component should extend or subclass the Component class. For more about E6 classes and inheritence, please see our tutorial on Understanding Classes in JavaScript.

      Add the following code to create a React component that extends react.Component:

      ~/djangoreactproject/frontend/src/CustomersList.js

      
      ...
      class  CustomersList  extends  Component {
      
          constructor(props) {
              super(props);
              this.state  = {
                  customers: [],
                  nextPageURL:  ''
              };
              this.nextPage  =  this.nextPage.bind(this);
              this.handleDelete  =  this.handleDelete.bind(this);
          }
      }
      export  default  CustomersList;
      

      Inside the constructor, we are initializing the state object. This holds the state variables of our component using an empty customers array. This array will hold customers and a nextPageURL that will hold the URL of the next page to retrieve from the back-end API. We are also binding the nextPage() and handleDelete() methods to this so they will be accessible from the HTML code.

      Next, add the componentDidMount() method and a call to getCustomers() within the CustomersList class, before the closing curly brace.

      The componentDidMount() method is a lifecycle method of the component that is called when the component is created and inserted into the DOM. getCustomers() calls the Customers Service object to get the first page of data and the link of the next page from the Django backend:

      ~/djangoreactproject/frontend/src/CustomersList.js

      
      ...
      componentDidMount() {
          var  self  =  this;
          customersService.getCustomers().then(function (result) {
              self.setState({ customers:  result.data, nextPageURL:  result.nextlink})
          });
      }
      

      Now add the handleDelete() method, which handles deleting a customer, below componentDidMount():

      ~/djangoreactproject/frontend/src/CustomersList.js

      
      ...
      handleDelete(e,pk){
          var  self  =  this;
          customersService.deleteCustomer({pk :  pk}).then(()=>{
              var  newArr  =  self.state.customers.filter(function(obj) {
                  return  obj.pk  !==  pk;
              });
              self.setState({customers:  newArr})
          });
      }
      

      The handleDelete() method calls the deleteCustomer() method to delete a customer using its pk (primary key). If the operation is successful, the customers array is filtered out for the removed customer.

      Next, add a nextPage() method to get the data for the next page and update the next page link:

      ~/djangoreactproject/frontend/src/CustomersList.js

      
      ...
      nextPage(){
          var  self  =  this;
          customersService.getCustomersByURL(this.state.nextPageURL).then((result) => {
              self.setState({ customers:  result.data, nextPageURL:  result.nextlink})
          });
      }
      

      The nextPage() method calls a getCustomersByURL() method, which takes the next page URL from the state object, this.state.nextPageURL, and updates the customers array with the returned data.

      Finally, add the component render() method, which renders a table of customers from the component state:

      ~/djangoreactproject/frontend/src/CustomersList.js

      
      ...
      render() {
      
          return (
          <div  className="customers--list">
              <table  className="table">
                  <thead  key="thead">
                  <tr>
                      <th>#</th>
                      <th>First Name</th>
                      <th>Last Name</th>
                      <th>Phone</th>
                      <th>Email</th>
                      <th>Address</th>
                      <th>Description</th>
                      <th>Actions</th>
                  </tr>
                  </thead>
                  <tbody>
                      {this.state.customers.map( c  =>
                      <tr  key={c.pk}>
                          <td>{c.pk}  </td>
                          <td>{c.first_name}</td>
                          <td>{c.last_name}</td>
                          <td>{c.phone}</td>
                          <td>{c.email}</td>
                          <td>{c.address}</td>
                          <td>{c.description}</td>
                          <td>
                          <button  onClick={(e)=>  this.handleDelete(e,c.pk) }> Delete</button>
                          <a  href={"/customer/" + c.pk}> Update</a>
                          </td>
                      </tr>)}
                  </tbody>
              </table>
              <button  className="btn btn-primary"  onClick=  {  this.nextPage  }>Next</button>
          </div>
          );
      }
      

      This is the full content of the file:

      ~/djangoreactproject/frontend/src/CustomersList.js

      import  React, { Component } from  'react';
      import  CustomersService  from  './CustomersService';
      
      const  customersService  =  new  CustomersService();
      
      class  CustomersList  extends  Component {
      
      constructor(props) {
          super(props);
          this.state  = {
              customers: [],
              nextPageURL:  ''
          };
          this.nextPage  =  this.nextPage.bind(this);
          this.handleDelete  =  this.handleDelete.bind(this);
      }
      
      componentDidMount() {
          var  self  =  this;
          customersService.getCustomers().then(function (result) {
              console.log(result);
              self.setState({ customers:  result.data, nextPageURL:  result.nextlink})
          });
      }
      handleDelete(e,pk){
          var  self  =  this;
          customersService.deleteCustomer({pk :  pk}).then(()=>{
              var  newArr  =  self.state.customers.filter(function(obj) {
                  return  obj.pk  !==  pk;
              });
      
              self.setState({customers:  newArr})
          });
      }
      
      nextPage(){
          var  self  =  this;
          console.log(this.state.nextPageURL);        
          customersService.getCustomersByURL(this.state.nextPageURL).then((result) => {
              self.setState({ customers:  result.data, nextPageURL:  result.nextlink})
          });
      }
      render() {
      
          return (
              <div  className="customers--list">
                  <table  className="table">
                  <thead  key="thead">
                  <tr>
                      <th>#</th>
                      <th>First Name</th>
                      <th>Last Name</th>
                      <th>Phone</th>
                      <th>Email</th>
                      <th>Address</th>
                      <th>Description</th>
                      <th>Actions</th>
                  </tr>
                  </thead>
                  <tbody>
                  {this.state.customers.map( c  =>
                      <tr  key={c.pk}>
                      <td>{c.pk}  </td>
                      <td>{c.first_name}</td>
                      <td>{c.last_name}</td>
                      <td>{c.phone}</td>
                      <td>{c.email}</td>
                      <td>{c.address}</td>
                      <td>{c.description}</td>
                      <td>
                      <button  onClick={(e)=>  this.handleDelete(e,c.pk) }> Delete</button>
                      <a  href={"/customer/" + c.pk}> Update</a>
                      </td>
                  </tr>)}
                  </tbody>
                  </table>
                  <button  className="btn btn-primary"  onClick=  {  this.nextPage  }>Next</button>
              </div>
              );
        }
      }
      export  default  CustomersList;
      

      Now that we've created the CustomersList component for displaying the list of customers, we can add the component that handles customer creation and updates.

      Step 8 — Adding the Customer Create and Update React Component

      In this step, we'll create the CustomerCreateUpdate component, which will handle creating and updating customers. It will do this by providing a form that users can use to either enter data about a new customer or update an existing entry.

      In frontend/src, create a CustomerCreateUpdate.js file:

      • nano ~/djangoreactproject/frontend/src/CustomerCreateUpdate.js

      Add the following code to create a React component, importing React and Component:

      ~/djangoreactproject/frontend/src/CustomerCreateUpdate.js

      import  React, { Component } from  'react';
      

      We can also import and instantiate the CustomersService class we created in the previous step, which provides methods that interface with the REST API backend:

      ~/djangoreactproject/frontend/src/CustomerCreateUpdate.js

      ...
      import  CustomersService  from  './CustomersService';
      
      const  customersService  =  new  CustomersService();
      

      Next, create a CustomerCreateUpdate component that extends Component to create and update customers:

      ~/djangoreactproject/frontend/src/CustomerCreateUpdate.js

      
      ...
      class  CustomerCreateUpdate  extends  Component {
      
          constructor(props) {
              super(props);
          }
      
      }
      export default CustomerCreateUpdate;
      

      Within the class definition, add the render() method of the component, which renders an HTML form that takes information about the customer:

      ~/djangoreactproject/frontend/src/CustomerCreateUpdate.js

      
      ...
      render() {
              return (
                <form onSubmit={this.handleSubmit}>
                <div className="form-group">
                  <label>
                    First Name:</label>
                    <input className="form-control" type="text" ref='firstName' />
      
                  <label>
                    Last Name:</label>
                    <input className="form-control" type="text" ref='lastName'/>
      
                  <label>
                    Phone:</label>
                    <input className="form-control" type="text" ref='phone' />
      
                  <label>
                    Email:</label>
                    <input className="form-control" type="text" ref='email' />
      
                  <label>
                    Address:</label>
                    <input className="form-control" type="text" ref='address' />
      
                  <label>
                    Description:</label>
                    <textarea className="form-control" ref='description' ></textarea>
      
      
                  <input className="btn btn-primary" type="submit" value="Submit" />
                  </div>
                </form>
              );
        }
      

      For each form input element, the method adds a ref property to access and set the value of the form element.

      Next, above the render() method, define a handleSubmit(event) method so that you have the proper functionality when a user clicks on the submit button:

      ~/djangoreactproject/frontend/src/CustomerCreateUpdate.js

      
      ...
      handleSubmit(event) {
          const { match: { params } } =  this.props;
          if(params  &&  params.pk){
              this.handleUpdate(params.pk);
          }
          else
          {
              this.handleCreate();
          }
          event.preventDefault();
      }
      
      ...
      

      The handleSubmit(event) method handles the form submission and, depending on the route, calls either the handleUpdate(pk) method to update the customer with the passed pk, or the handleCreate() method to create a new customer. We will define these methods shortly.

      Back on the component constructor, bind the newly added handleSubmit() method to this so you can access it in your form:

      ~/djangoreactproject/frontend/src/CustomerCreateUpdate.js

      ...
      class CustomerCreateUpdate extends Component {
      
      constructor(props) {
          super(props);
          this.handleSubmit = this.handleSubmit.bind(this);
      }
      ...
      

      Next, define the handleCreate() method to create a customer from the form data. Above the handleSubmit(event) method, add the following code:

      ~/djangoreactproject/frontend/src/CustomerCreateUpdate.js

      
      ...
      handleCreate(){
          customersService.createCustomer(
              {
              "first_name":  this.refs.firstName.value,
              "last_name":  this.refs.lastName.value,
              "email":  this.refs.email.value,
              "phone":  this.refs.phone.value,
              "address":  this.refs.address.value,
              "description":  this.refs.description.value
              }).then((result)=>{
                      alert("Customer created!");
              }).catch(()=>{
                      alert('There was an error! Please re-check your form.');
              });
      }
      
      ...
      

      The handleCreate() method will be used to create a customer from inputted data. It calls the corresponding CustomersService.createCustomer() method that makes the actual API call to the backend to create a customer.

      Next, below the handleCreate() method, define the handleUpdate(pk) method to implement updates:

      ~/djangoreactproject/frontend/src/CustomerCreateUpdate.js

      
      ...
      handleUpdate(pk){
      customersService.updateCustomer(
          {
          "pk":  pk,
          "first_name":  this.refs.firstName.value,
          "last_name":  this.refs.lastName.value,
          "email":  this.refs.email.value,
          "phone":  this.refs.phone.value,
          "address":  this.refs.address.value,
          "description":  this.refs.description.value
          }
          ).then((result)=>{
      
              alert("Customer updated!");
          }).catch(()=>{
              alert('There was an error! Please re-check your form.');
          });
      }
      

      The updateCustomer() method will update a customer by pk using the new information from the customer information form. It calls the customersService.updateCustomer() method.

      Next, add a componentDidMount() method. If the the user visits a customer/:pk route, we want to fill the form with information related to the customer using the primary key from the URL. To do that, we can add the getCustomer(pk) method after the component gets mounted in the lifecycle event of componentDidMount(). Add the following code below the component constructor to add this method:

      ~/djangoreactproject/frontend/src/CustomerCreateUpdate.js

      
      ...
      componentDidMount(){
          const { match: { params } } =  this.props;
          if(params  &&  params.pk)
          {
              customersService.getCustomer(params.pk).then((c)=>{
                  this.refs.firstName.value  =  c.first_name;
                  this.refs.lastName.value  =  c.last_name;
                  this.refs.email.value  =  c.email;
                  this.refs.phone.value  =  c.phone;
                  this.refs.address.value  =  c.address;
                  this.refs.description.value  =  c.description;
              })
          }
      }
      

      This is the full content of the file:

      ~/djangoreactproject/frontend/src/CustomerCreateUpdate.js

      import React, { Component } from 'react';
      import CustomersService from './CustomersService';
      
      const customersService = new CustomersService();
      
      class CustomerCreateUpdate extends Component {
          constructor(props) {
              super(props);
      
              this.handleSubmit = this.handleSubmit.bind(this);
            }
      
            componentDidMount(){
              const { match: { params } } = this.props;
              if(params && params.pk)
              {
                customersService.getCustomer(params.pk).then((c)=>{
                  this.refs.firstName.value = c.first_name;
                  this.refs.lastName.value = c.last_name;
                  this.refs.email.value = c.email;
                  this.refs.phone.value = c.phone;
                  this.refs.address.value = c.address;
                  this.refs.description.value = c.description;
                })
              }
            }
      
            handleCreate(){
              customersService.createCustomer(
                {
                  "first_name": this.refs.firstName.value,
                  "last_name": this.refs.lastName.value,
                  "email": this.refs.email.value,
                  "phone": this.refs.phone.value,
                  "address": this.refs.address.value,
                  "description": this.refs.description.value
              }          
              ).then((result)=>{
                alert("Customer created!");
              }).catch(()=>{
                alert('There was an error! Please re-check your form.');
              });
            }
            handleUpdate(pk){
              customersService.updateCustomer(
                {
                  "pk": pk,
                  "first_name": this.refs.firstName.value,
                  "last_name": this.refs.lastName.value,
                  "email": this.refs.email.value,
                  "phone": this.refs.phone.value,
                  "address": this.refs.address.value,
                  "description": this.refs.description.value
              }          
              ).then((result)=>{
                console.log(result);
                alert("Customer updated!");
              }).catch(()=>{
                alert('There was an error! Please re-check your form.');
              });
            }
            handleSubmit(event) {
              const { match: { params } } = this.props;
      
              if(params && params.pk){
                this.handleUpdate(params.pk);
              }
              else
              {
                this.handleCreate();
              }
      
              event.preventDefault();
            }
      
            render() {
              return (
                <form onSubmit={this.handleSubmit}>
                <div className="form-group">
                  <label>
                    First Name:</label>
                    <input className="form-control" type="text" ref='firstName' />
      
                  <label>
                    Last Name:</label>
                    <input className="form-control" type="text" ref='lastName'/>
      
                  <label>
                    Phone:</label>
                    <input className="form-control" type="text" ref='phone' />
      
                  <label>
                    Email:</label>
                    <input className="form-control" type="text" ref='email' />
      
                  <label>
                    Address:</label>
                    <input className="form-control" type="text" ref='address' />
      
                  <label>
                    Description:</label>
                    <textarea className="form-control" ref='description' ></textarea>
      
      
                  <input className="btn btn-primary" type="submit" value="Submit" />
                  </div>
                </form>
              );
            }  
      }
      
      export default CustomerCreateUpdate;
      

      With the CustomerCreateUpdate component created, we can update the main App component to add links to the different components we've created.

      Step 9 — Updating the Main App Component

      In this section, we'll update the App component of our application to create links to the components we've created in the previous steps.

      From the frontend folder, run the following command to install the React Router, which allows you to add routing and navigation between various React components:

      • cd ~/djangoreactproject/frontend
      • npm install --save react-router-dom

      Next, open ~/djangoreactproject/frontend/src/App.js:

      • nano ~/djangoreactproject/frontend/src/App.js

      Delete everything that's there and add the following code to import the necessary classes for adding routing. These include BrowserRouter, which creates a Router component, and Route, which creates a route component:

      ~/djangoreactproject/frontend/src/App.js

      import  React, { Component } from  'react';
      import { BrowserRouter } from  'react-router-dom'
      import { Route, Link } from  'react-router-dom'
      import  CustomersList  from  './CustomersList'
      import  CustomerCreateUpdate  from  './CustomerCreateUpdate'
      import  './App.css';
      

      BrowserRouter keeps the UI in sync with the URL using the HTML5 history API.

      Next, create a base layout that provides the base component to be wrapped by the BrowserRouter component:

      ~/djangoreactproject/frontend/src/App.js

      ...
      
      const  BaseLayout  = () => (
      <div  className="container-fluid">
          <nav  className="navbar navbar-expand-lg navbar-light bg-light">
              <a  className="navbar-brand"  href="#">Django React Demo</a>
              <button  className="navbar-toggler"  type="button"  data-toggle="collapse"  data-target="#navbarNavAltMarkup"  aria-controls="navbarNavAltMarkup"  aria-expanded="false"  aria-label="Toggle navigation">
              <span  className="navbar-toggler-icon"></span>
          </button>
          <div  className="collapse navbar-collapse"  id="navbarNavAltMarkup">
              <div  className="navbar-nav">
                  <a  className="nav-item nav-link"  href="/">CUSTOMERS</a>
                  <a  className="nav-item nav-link"  href="http://www.digitalocean.com/customer">CREATE CUSTOMER</a>
              </div>
          </div>
          </nav>
          <div  className="content">
              <Route  path="/"  exact  component={CustomersList}  />
              <Route  path="/customer/:pk"  component={CustomerCreateUpdate}  />
              <Route  path="/customer/"  exact  component={CustomerCreateUpdate}  />
          </div>
      </div>
      )
      

      We use the Route component to define the routes of our application; the component the router should load once a match is found. Each route needs a path to specify the path to be matched and a component to specify the component to load. The exact property tells the router to match the exact path.

      Finally, create the App component, the root or top-level component of our React application:

      ~/djangoreactproject/frontend/src/App.js

      ...
      
      class  App  extends  Component {
      
      render() {
          return (
          <BrowserRouter>
              <BaseLayout/>
          </BrowserRouter>
          );
      }
      }
      export  default  App;
      

      We have wrapped the BaseLayout component with the BrowserRouter component since our app is meant to run in the browser.

      The completed file looks like this:

      ~/djangoreactproject/frontend/src/App.js

      import React, { Component } from 'react';
      import { BrowserRouter } from 'react-router-dom'
      import { Route, Link } from 'react-router-dom'
      
      import  CustomersList from './CustomersList'
      import  CustomerCreateUpdate  from './CustomerCreateUpdate'
      import './App.css';
      
      const BaseLayout = () => (
        <div className="container-fluid">
      <nav className="navbar navbar-expand-lg navbar-light bg-light">
        <a className="navbar-brand" href="#">Django React Demo</a>
        <button className="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarNavAltMarkup" aria-controls="navbarNavAltMarkup" aria-expanded="false" aria-label="Toggle navigation">
          <span className="navbar-toggler-icon"></span>
        </button>
        <div className="collapse navbar-collapse" id="navbarNavAltMarkup">
          <div className="navbar-nav">
            <a className="nav-item nav-link" href="/">CUSTOMERS</a>
            <a className="nav-item nav-link" href="http://www.digitalocean.com/customer">CREATE CUSTOMER</a>
      
          </div>
        </div>
      </nav>  
      
          <div className="content">
            <Route path="/" exact component={CustomersList} />
            <Route path="/customer/:pk"  component={CustomerCreateUpdate} />
            <Route path="/customer/" exact component={CustomerCreateUpdate} />
      
          </div>
      
        </div>
      )
      
      class App extends Component {
        render() {
          return (
            <BrowserRouter>
              <BaseLayout/>
            </BrowserRouter>
          );
        }
      }
      
      export default App;
      

      After adding routing to our application, we are now ready to test the application. Navigate to http://localhost:3000. You should see the first page of the application:

      Application Home Page

      With this application in place, you now have the base for a CRM application.

      Conclusion

      In this tutorial, you created a demo application using Django and React. You used the Django REST framework to build the REST API, Axios to consume the API, and Bootstrap 4 to style your CSS. You can find the source code of this project in this GitHub repository.

      This tutorial setup used separate front-end and back-end apps. For a different approach to integrating React with Django, check this tutorial and this tutorial.

      For more information about building an application with Django, you can follow the Django development series. You can also look at the official Django docs.



      Source link

      Como Fazer Scraping em Páginas Web com Beautiful Soup and Python 3


      Introdução

      Muitos projetos de análise de dados, big data, e aprendizado de máquina exigem o scraping de websites para coletar os dados com os quais você irá trabalhar. A linguagem de programação Python é largamente utilizada na comunidade de data science, e, portanto, tem um ecossistema de módulos e ferramentas que você pode usar em seus próprios projetos. Neste tutorial estaremos nos concentrando no módulo Beautiful Soup.

      Beautiful Soup, uma alusão à música Mock Turtle’s encontrada no Capítulo 10 de Alice no País das Maravilhas, de Lewis Carroll, é uma biblioteca do Python que permite um retorno rápido em projetos de web scraping. Atualmente disponível como Beautiful Soup 4 e compatível tanto com Python 2.7 quanto com Python 3, o Beautiful Soup cria uma árvore de análise a partir de documentos HTML e XML analisados (incluindo documentos com tags não fechadas ou tag soup e outras marcações malformadas).

      Neste tutorial, iremos coletar e analisar uma página web de forma a pegar dados textuais e gravar as informações que tivermos recolhido em um arquivo CSV.

      Pré-requisitos

      Antes de trabalhar com este tutorial, você deve ter um ambiente de programação Python local ou baseado em servidor configurado em sua máquina.

      Você deve ter os módulos Requests e Beautiful Soup instalados, o que pode ser conseguido seguindo o nosso tutorial “How To Work with Web Data Using Requests and Beautiful Soup with Python 3.” Também seria útil ter familiaridade no trabalho com esses módulos.

      Adicionalmente, uma vez que vamos trabalhar com dados extraídos da web, você deve estar confortável com a estrutura e a marcação de tags HTML.

      Entendendo os Dados

      Neste tutorial, iremos trabalhar com dados do site oficial do National Gallery of Art nos Estados Unidos. O National Gallery é um museu de arte localizado no National Mall em Washington, D.C. Ele possui mais de 120.000 peças datadas desde o Renascimento aos dias atuais feitas por mais de 13.000 artistas.

      Gostaríamos de pesquisar o Índice de Artistas, que, no momento da atualização deste tutorial, estava disponível via Internet Archive’s Wayback Machine na seguinte URL:

      https://web.archive.org/web/20170131230332/https://www.nga.gov/collection/an.shtm

      Nota: A longa URL acima é devido a este site ter sido arquivado pelo Internet Archive.

      O Internet Archive é uma biblioteca digital sem fins lucrativos que fornece acesso livre a sites da internet e outras mídias digitais. A organização tira instantâneos de websites para preservar a história dos sites, e atualmente, podemos acessar uma versão mais antiga do site da National Gallery que estava disponível quando este tutorial foi escrito pela primeira vez. O Internet Archive é uma boa ferramenta para se ter em mente sempre que estiver fazendo qualquer tipo de scraping de dados históricos, incluindo a comparação entre iterações do mesmo site e dados disponíveis.

      Logo abaixo do cabeçalho do Internet Archive, você verá uma página como esta:

      Como estamos fazendo esse projeto para aprender sobre o web scraping com o Beautiful Soup, não precisamos extrair muitos dados do site, por isso, vamos limitar o escopo dos dados do artista que estamos tentando capturar. Vamos, portanto, escolher uma letra — em nosso exemplo escolheremos a letra Z — e veremos uma página como esta:

      Na página acima, vemos que o primeiro artista listado no momento da escrita é Zabaglia, Niccola, o que é uma boa coisa a se notar quando começamos a extrair dados. Começaremos trabalhando com essa primeira página, com a seguinte URL para a letra Z:

      https://web.archive.org/web/20121007172955/http://www.nga.gov/collection/anZ1.htm

      É importante observar, para análise posterior, quantas páginas existem para a letra que você está escolhendo listar, o que você pode descobrir clicando na última página de artistas. Nesse caso, existe um total de 4 páginas, e o último artista listado no momento da escrita é Zykmund, Václav. A última página de artistas com Z tem a seguinte URL:

      https://web.archive.org/web/20121010201041/http://www.nga.gov/collection/anZ4.htm

      Contudo, você também pode acessar a página acima usando a mesma string numérica do Internet Archive da primeira página:

      https://web.archive.org/web/20121007172955/http://www.nga.gov/collection/anZ4.htm

      É importante observar isso, pois mais adiante neste tutorial faremos a iteração dessas páginas.

      Para começar a se familiarizar com a forma que essa página web é configurada, você pode dar uma olhada em seu DOM, que o ajudará a entender como o HTML é estruturado. Para inspecionar o DOM, você pode abrir Ferramentas do Desenvolvedor do seu navegador.

      Importando as Bibliotecas

      Para iniciar nosso projeto de codificação, vamos ativar nosso ambiente de programação. Certifique-se de que você está no diretório onde o seu ambiente de desenvolvimento está localizado, e execute o seguinte comando.

      Com o nosso ambiente de programação ativado, vamos criar um novo arquivo, com o nano por exemplo. Você pode nomear seu arquivo como quiser, vamos chamá-lo de nga_z_artists.py nesse tutorial.

      Nesse arquivo, podemos começar a importar as bibliotecas que iremos utilizar — Requests e Beautiful Soup.

      A biblioteca Requests lhe permite fazer uso do HTTP dentro dos seus programas Python em um formato legível, e o módulo Beautiful Soup é projetado para fazer web scraping rapidamente.

      Vamos importar tanto o Requests quanto o Beautiful Soup com a declaração import. Para o Beautiful Soup iremos importá-lo do bs4, o pacote no qual o Beautiful Soup 4 é encontrado.

      nga_z_artists.py

      
      # Importar bibliotecas
      import requests
      from bs4 import BeautifulSoup
      

      Com os módulos Requests e Beautiful Soup importados, podemos passar a trabalhar para coletar primeiro uma página e analisá-la.

      Coletando e Analisando uma Página Web

      O próximo passo que precisaremos fazer é coletar a URL da primeira página web com o Requests. Iremos atribuir a URL da primeira página à variável page usando o método requests.get().

      nga_z_artists.py

      
      import requests
      from bs4 import BeautifulSoup
      
      
      # Coletar a primeira página da lista de artistas
      page = requests.get('https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ1.htm')
      
      

      Nota: Como a URL é longa, o código acima, bem como todo este tutorial não passarão no PEP 8 E501, que sinaliza linhas com mais de 79 caracteres. Você pode querer atribuir a URL a uma variável para tornar o código mais legível nas versões finais. O código neste tutorial é para fins de demonstração e permitirá que você troque URLs mais curtas como parte de seus próprios projetos.

      Agora iremos criar o objeto BeautifulSoup, ou uma árvore de análise. Esse objeto utiliza como argumento o documento page.text do Requests (o conteúdo da resposta do servidor) e então o analisa através do html.parser interno do Python.

      nga_z_artists.py

      
      import requests
      from bs4 import BeautifulSoup
      
      
      page = requests.get('https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ1.htm')
      
      # Criar o objeto BeautifulSoup
      soup = BeautifulSoup(page.text, 'html.parser')
      
      

      Com a nossa página coletada, analisada e configurada como um objeto BeautifulSoup, podemos passar para a coleta dos dados que gostaríamos.

      Pegando Texto de uma Página Web

      Para este projeto, iremos coletar nomes de artistas e os links relevantes disponíveis no website. Você pode querer coletar dados diferentes, tais como a nacionalidade dos artistas e datas. Para quaisquer dados que você queira coletar, você precisa descobrir como ele é descrito pelo DOM da página web.

      Para fazer isso, no seu navegador web, clique com o botão direito — ou CTRL + clique no macOS — no nome do primeiro artista, Zabaglia, Niccola. Dentro do menu de contexto que aparece, você deve ver um item de menu semelhante ao Inspecionar Elemento (Firefox) ou Inspecionar (Chrome).

      Após clicar no item de menu relevante Inspecionar, as ferramentas para desenvolvedores web devem aparecer no seu navegador. Queremos procurar pela classe e as tags associadas aos nomes dos artistas nessa lista.

      Veremos primeiro que a tabela de nomes está dentro de tags <div> onde class="BodyText". É importante observar isso, para que só procuremos texto nessa seção da página web. Também notamos que o nome Zabaglia, Niccola está em uma tag de link, já que o nome faz referência a uma página web que descreve o artista. Então, vamos querer referenciar a tag <a> para links. O nome de cada artista é uma referência a um link.

      Para fazer isso, iremos utilizar os métodos find() e find_all() do Beautiful Soup a fim de extrair o texto dos nomes dos artistas do BodyText <div>.

      nga_z_artists.py

      
      import requests
      from bs4 import BeautifulSoup
      
      
      # Coletar e analisar a primeira página
      page = requests.get('https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ1.htm')
      soup = BeautifulSoup(page.text, 'html.parser')
      
      # Pegar todo o texto da div BodyText
      artist_name_list = soup.find(class_='BodyText')
      
      # Pegar o texto de todas as instâncias da tag <a> dentro da div BodyText
      artist_name_list_items = artist_name_list.find_all('a')
      
      

      A seguir, na parte inferior do nosso arquivo de programa, criaremos um loop for para iterar todos os nomes de artistas que acabamos de colocar na variável artist_name_list_items.

      Vamos imprimir esses nomes com o método prettify() para transformar a árvore de análise do Beautiful Soup em uma string Unicode bem formatada.

      nga_z_artists.py

      
      ...
      artist_name_list = soup.find(class_='BodyText')
      artist_name_list_items = artist_name_list.find_all('a')
      
      # Criar loop para imprimir todos os nomes de artistas
      for artist_name in artist_name_list_items:
          print(artist_name.prettify())
      
      

      Vamos executar o programa como ele está até agora:

      Assim que fizermos isso, receberemos a seguinte saída:

      Output

      <a href="http://www.digitalocean.com/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=11630"> Zabaglia, Niccola </a> ... <a href="http://www.digitalocean.com/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=3427"> Zao Wou-Ki </a> <a href="http://www.digitalocean.com/web/20121007172955/https://www.nga.gov/collection/anZ2.htm"> Zas-Zie </a> <a href="http://www.digitalocean.com/web/20121007172955/https://www.nga.gov/collection/anZ3.htm"> Zie-Zor </a> <a href="http://www.digitalocean.com/web/20121007172955/https://www.nga.gov/collection/anZ4.htm"> <strong> next <br/> page </strong> </a>

      O que vemos na saída nesse ponto é o texto completo e as tags relativas a todos os nomes de artistas dentro de tags <a> encontradas na tag <div class="BodyText"> na primeira página, bem como algum texto de link adicional na parte inferior. Como não queremos essa informação extra, vamos trabalhar para remover isso na próxima seção.

      Removendo Dados Supérfluos

      Até agora, conseguimos coletar todos os dados de texto do link dentro de uma seção <div> da nossa página web. No entanto, não queremos ter os links inferiores que não fazem referência aos nomes dos artistas. Por isso, vamos trabalhar para remover essa parte.

      Para remover os links inferiores da página, vamos clicar novamente com o botão direito e Inspecionar o DOM. Veremos que os links na parte inferior da seção <div class="BodyText"> estão contidos em uma tabela HTML: <table class="AlphaNav">:

      Podemos, portanto, usar o Beautiful Soup para encontrar a classe AlphaNav e usar o método decompose() para remover uma tag da árvore de análise e depois destruí-la juntamente com seu conteúdo.

      Usaremos a variável last_links para fazer referência a esses links inferiores e adicioná-los ao arquivo do programa:

      nga_z_artists.py

      
      import requests
      from bs4 import BeautifulSoup
      
      
      page = requests.get('https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ1.htm')
      
      soup = BeautifulSoup(page.text, 'html.parser')
      
      # Remover links inferiores
      last_links = soup.find(class_='AlphaNav')
      last_links.decompose()
      
      artist_name_list = soup.find(class_='BodyText')
      artist_name_list_items = artist_name_list.find_all('a')
      
      for artist_name in artist_name_list_items:
          print(artist_name.prettify())
      
      

      Agora, quando executarmos o programa com o comando python nga_z_artist.py, receberemos a seguinte saída:

      Output

      <a href="http://www.digitalocean.com/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=11630"> Zabaglia, Niccola </a> <a href="http://www.digitalocean.com/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=34202"> Zaccone, Fabian </a> ... <a href="http://www.digitalocean.com/web/20121007172955/http://www.nga.gov/cgi-bin/tsearch?artistid=11631"> Zanotti, Giampietro </a> <a href="http://www.digitalocean.com/web/20121007172955/http://www.nga.gov/cgi-bin/tsearch?artistid=3427"> Zao Wou-Ki </a>

      Nesse ponto, vemos que a saída não inclui mais os links na parte inferior da página web e agora exibe apenas os links associados aos nomes dos artistas.

      Até agora, focamos especificamente os links com os nomes dos artistas, mas temos os dados de tags extras que realmente não queremos. Vamos remover isso na próxima seção.

      Pegando o Conteúdo de uma Tag

      Para acessar apenas os nomes reais dos artistas, queremos focar no conteúdo das tags <a> em vez de imprimir toda a tag de link.

      Podemos fazer isso com o .contents do Beautiful Soup, que irá retornar a tag filha com um tipo de dados lista do Python.

      Vamos revisar o loop for para que, em vez de imprimir o link inteiro e sua tag, façamos a impressão da lista das filhas (ou seja, os nomes completos dos artistas).

      nga_z_artists.py

      
      import requests
      from bs4 import BeautifulSoup
      
      
      page = requests.get('https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ1.htm')
      
      soup = BeautifulSoup(page.text, 'html.parser')
      
      last_links = soup.find(class_='AlphaNav')
      last_links.decompose()
      
      artist_name_list = soup.find(class_='BodyText')
      artist_name_list_items = artist_name_list.find_all('a')
      
      # Usar .contents para pegar as tags <a> filhas
      for artist_name in artist_name_list_items:
          names = artist_name.contents[0]
          print(names)
      
      

      Note que estamos iterando na lista acima chamando o número do índice de cada item.

      Podemos executar o programa com o comando python para ver a seguinte saída:

      Output

      Zabaglia, Niccola Zaccone, Fabian Zadkine, Ossip ... Zanini-Viola, Giuseppe Zanotti, Giampietro Zao Wou-Ki

      Recebemos de volta uma lista de todos os nomes dos artistas disponíveis na primeira página da letra Z.

      Mas, e se quisermos também capturar as URLs associadas a esses artistas? Podemos extrair URLs encontradas dentro de tags <a> utilizando o método get('href') do Beautiful Soup.

      A partir da saída dos links acima, sabemos que a URL inteira não está sendo capturada, então vamos concatenar a string do link com o início da string da URL (nesse caso https://web.archive.org/).

      Estas linhas também serão adicionadas ao loop for:

      nga_z_artists.py

      
      ...
      for artist_name in artist_name_list_items:
          names = artist_name.contents[0]
          links = 'https://web.archive.org' + artist_name.get('href')
          print(names)
          print(links)
      
      

      Quando executamos o programa acima, receberemos tanto os nomes dos artistas quanto as URLs para os links que nos dizem mais sobre eles:

      Output

      Zabaglia, Niccola https://web.archive.org/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=11630 Zaccone, Fabian https://web.archive.org/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=34202 ... Zanotti, Giampietro https://web.archive.org/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=11631 Zao Wou-Ki https://web.archive.org/web/20121007172955/https://www.nga.gov/cgi-bin/tsearch?artistid=3427

      Embora estejamos agora recebendo informações do site, ele está apenas imprimindo em nossa janela de terminal. Em vez disso, vamos capturar esses dados para podermos usá-los em outro lugar, gravando-os em um arquivo.

      Gravando os Dados em um Arquivo CSV

      Coletar dados que só residem em uma janela de terminal não é muito útil. Arquivos de valores separados por vírgulas (CSV) nos permitem armazenar dados tabulares em texto plano, e é um formato comum para planilhas e bancos de dados. Antes de iniciar esta seção, você deve familiarizar-se com como manipular arquivos de texto sem formatação em Python.

      Primeiro, precisamos importar o módulo interno csv do Python junto com os outros módulos no topo do arquivo de código:

      import csv
      

      Em seguida, vamos criar e abrir um arquivo chamado z-artist-names.csv para que possamos gravar nele (iremos utilizar aqui a variável f para o arquivo), utilizando o modo 'w'. Também vamos escrever os cabeçalhos da primeira linha: Name and Link que iremos passar para o método writerow() como uma lista.

      f = csv.writer(open('z-artist-names.csv', 'w'))
      f.writerow(['Name', 'Link'])
      

      Finalmente, dentro do nosso loop for, vamos escrever cada linha com os names ou nomes dos artistas e seus links associados:

      f.writerow([names, links])
      

      Você pode ver as linhas para cada uma dessas tarefas no arquivo abaixo:

      nga_z_artists.py

      
      import requests
      import csv
      from bs4 import BeautifulSoup
      
      
      page = requests.get('https://web.archive.org/web/20121007172955/http://www.nga.gov/collection/anZ1.htm')
      
      soup = BeautifulSoup(page.text, 'html.parser')
      
      last_links = soup.find(class_='AlphaNav')
      last_links.decompose()
      
      # Criar um arquivo para gravar, adicionar linha de cabeçalhos
      f = csv.writer(open('z-artist-names.csv', 'w'))
      f.writerow(['Name', 'Link'])
      
      artist_name_list = soup.find(class_='BodyText')
      artist_name_list_items = artist_name_list.find_all('a')
      
      for artist_name in artist_name_list_items:
          names = artist_name.contents[0]
          links = 'https://web.archive.org' + artist_name.get('href')
      
      
          # Adicionar em uma linha o nome de cada artista e o link associado
          f.writerow([names, links])
      
      

      Quando você executar o programa agora com o comando python, nenhuma saída será retornada para sua janela de terminal. Em vez disso, um arquivo será criado no diretório em que você está trabalhando, chamado z-artist-names.csv.

      Dependendo do que você usa para abrí-lo, ele deve ser algo assim:

      z-artist-names.csv

      
      Name,Link
      "Zabaglia, Niccola",https://web.archive.org/web/20121007172955/http://www.nga.gov/cgi-bin/tsearch?artistid=11630
      "Zaccone, Fabian",https://web.archive.org/web/20121007172955/http://www.nga.gov/cgi-bin/tsearch?artistid=34202
      "Zadkine, Ossip",https://web.archive.org/web/20121007172955/http://www.nga.gov/cgi-bin/tsearch?artistid=3475w
      ...
      
      

      Ou, ele pode se parecer mais com uma planilha:

      Em ambos os casos, agora você pode usar esse arquivo para trabalhar com os dados de maneiras mais significativas, já que as informações coletadas agora estão armazenadas no disco do seu computador.

      Recuperando Páginas Relacionadas

      Criamos um programa que extrairá dados da primeira página da lista de artistas cujos sobrenomes começam com a letra Z. Porém, existem 4 páginas desses artistas no total, disponíveis no website.

      Para coletar todas essas páginas, podemos executar mais iterações com loops for. Isso revisará a maior parte do código que escrevemos até agora, mas empregará conceitos semelhantes.

      Para começar, vamos inicializar uma lista para manter as páginas:

      pages = []
      

      Vamos preencher essa lista inicializada com o seguinte loop for:

      for i in range(1, 5):
          url = 'https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ' + str(i) + '.htm'
          pages.append(url)
      
      

      Anteriormente neste tutorial, observamos que devemos prestar atenção ao número total de páginas que contêm nomes de artistas começando com a letra Z (ou qualquer letra que estivermos utilizando). Uma vez que existem 4 páginas para a letra Z, construímos o loop foracima com um intervalo de 1 a 5 de modo que ele vai iterar através de cada uma das 4 páginas.

      Para este website específico, as URLs começam com a string https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZe são seguidas com um número de página (que será o inteiro i do loop for que convertemos para uma string) e terminam com .htm. Iremos concatenar estas strings e depois acrescentar o resultado à lista pages.

      Além desse loop, teremos um segundo loop que passará por cada uma das páginas acima. O código nesse loop for será parecido com o código que criamos até agora, já que ele está executando a tarefa que completamos para a primeira página dos artistas com a letra Z para cada um das 4 páginas do total. Observe que, como colocamos o programa original no segundo loop for, agora temos o loop original como um loop for aninhado contido nele.

      Os dois loops for ficarão assim:

      pages = []
      
      for i in range(1, 5):
          url = 'https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ' + str(i) + '.htm'
          pages.append(url)
      
      for item in pages:
          page = requests.get(item)
          soup = BeautifulSoup(page.text, 'html.parser')
      
          last_links = soup.find(class_='AlphaNav')
          last_links.decompose()
      
          artist_name_list = soup.find(class_='BodyText')
          artist_name_list_items = artist_name_list.find_all('a')
      
          for artist_name in artist_name_list_items:
              names = artist_name.contents[0]
              links = 'https://web.archive.org' + artist_name.get('href')
      
              f.writerow([names, links])
      
      

      No código acima, você deve ver que o primeiro loop for está iterando nas páginas e o segundo loop for está extraindo dados de cada uma dessas páginas e, em seguida, adicionando os nomes e links dos artistas, linha por linha, em cada linha de cada página.

      Estes dois loops for estão abaixo das declaraçõs import, da criação e escrita do arquivo CSV (com a linha para a escrita dos cabeçalhos do arquivo), e a inicialização da variável pages (atribuída a uma lista).

      Dentro de um contexto macro do arquivo de programação, o código completo se parece com isto:

      nga_z_artists.py

      
      import requests
      import csv
      from bs4 import BeautifulSoup
      
      
      f = csv.writer(open('z-artist-names.csv', 'w'))
      f.writerow(['Name', 'Link'])
      
      pages = []
      
      for i in range(1, 5):
          url = 'https://web.archive.org/web/20121007172955/https://www.nga.gov/collection/anZ' + str(i) + '.htm'
          pages.append(url)
      
      
      for item in pages:
          page = requests.get(item)
          soup = BeautifulSoup(page.text, 'html.parser')
      
          last_links = soup.find(class_='AlphaNav')
          last_links.decompose()
      
          artist_name_list = soup.find(class_='BodyText')
          artist_name_list_items = artist_name_list.find_all('a')
      
          for artist_name in artist_name_list_items:
              names = artist_name.contents[0]
              links = 'https://web.archive.org' + artist_name.get('href')
      
              f.writerow([names, links])
      
      
      

      Como esse programa está fazendo um trabalho, levará algum tempo para criar o arquivo CSV. Depois de concluído, a saída estará comleta, mostrando os nomes dos artistas e seus links associados de Zabaglia, Niccola até Zykmund, Václav.

      Sendo Cuidadoso

      Ao fazer scraping em páginas web, é importante manter-se cuidadoso com os servidores dos quais você está pegando informações.

      Verifique se o site tem termos de serviço ou termos de uso relacionados ao web scraping. Além disso, verifique se o site tem uma API que permite coletar dados antes de você mesmo fazer scraping.

      Certifique-se de não acessar continuamente os servidores para coletar dados. Depois de coletar o que você precisa de um site, execute scripts que vasculhem pelos dados localmente, em vez de sobrecarregar os servidores de outra pessoa.

      Adicionalmente, é uma boa ideia fazer web scraping com um cabeçalho que tenha o seu nome e e-mail para que o website possa identificá-lo e fazer o acompanhamento caso tenha alguma dúvida. Um exemplo de cabeçalho que você pode usar com a biblioteca Requests do Python é o seguinte:

      import requests
      
      headers = {
          'User-Agent': 'Seu nome, example.com',
          'From': 'email@example.com'
      }
      
      url = 'https://example.com'
      
      page = requests.get(url, headers = headers)
      
      

      A utilização de cabeçalhos com informações identificáveis ​​garante que as pessoas que acessam os logs de um servidor possam entrar em contato com você.

      Conclusão

      Este tutorial usou o Python e o Beautiful Soup para coletar dados de um website. Armazenamos o texto que reunimos em um arquivo CSV.

      Você pode continuar trabalhando neste projeto coletando mais dados e tornando seu arquivo CSV mais robusto. Por exemplo, você pode querer incluir as nacionalidades e os anos de cada artista. Você também pode usar o que aprendeu para coletar dados de outros sites.

      Para continuar aprendendo sobre como extrair informações da web, leia nosso tutorial “How To Crawl A Web Page with Scrapy and Python 3.”



      Source link