One place for hosting & domains

      How To Build a Shopping Cart with Vue 3 and Vuex


      The author selected the Open Source Initiative to receive a donation as part of the Write for DOnations program.

      Introduction

      Vue.js is a performant and progressive Javascript framework. It is a popular framework on GitHub and has an active and helpful community.

      In order to show the capabilities of the Vue web framework, this tutorial will lead you through building the shopping cart of an e-commerce app. This app will store product information and hold the products that the customer wants to buy for checkout later. To store the information, you will explore a widely used state management library for Vue.js: Vuex. This will allow the shopping cart application to persist data to a server. You will also handle asynchronous task management using Vuex.

      Once you finish the tutorial, you will have a functioning shopping cart application like the following:

      Animation of user adding and deleting products from the shopping cart application

      Prerequisites

      Step 1 — Setting Up the Application with Vue CLI

      As of version 4.5.0, Vue CLI now provides a built-in option to choose the Vue 3 preset when creating a new project. The latest version of Vue CLI allows you to use Vue 3 out of the box and to update your existing Vue 2 project to Vue 3. In this step, you will use the Vue CLI to make your project, then install the front-end dependencies.

      First, install the latest version of Vue CLI by executing the following command from the terminal:

      This will install Vue CLI globally on your system.

      Note: On some systems, installing an npm package globally can result in a permission error, which will interrupt the installation. Since it is a security best practice to avoid using sudo with npm install, you can instead resolve this by changing npm’s default directory. If you encounter an EACCES error, follow the instructions at the official npm documentation.

      Check you have the right version with this command:

      You will get output like the following:

      Output

      @vue/cli 4.5.10

      Note: If you already have the older version of Vue CLI installed globally, execute the following command from the terminal to upgrade:

      Now, you can create a new project:

      • vue create vuex-shopping-cart

      This uses the Vue CLI command vue create to make a project named vuex-shopping-cart. For more information on the Vue CLI, check out How To Generate a Vue.js Single Page App With the Vue CLI.

      Next, you will receive the following prompt:

      Output

      Vue CLI v4.5.10 ? Please pick a preset: (Use arrow keys) ❯ Default ([Vue 2] babel, eslint) Default (Vue 3 Preview) ([Vue 3] babel, eslint) Manually select features

      Choose the Manually select features option from this list.

      Next, you will encounter the following prompt to customize your Vue app:

      Output

      ... ◉ Choose Vue version ◯ Babel ◯ TypeScript ◯ Progressive Web App (PWA) Support ◉ Router ◉ Vuex ◯ CSS Pre-processors ◯ Linter / Formatter ❯◯ Unit Testing ◯ E2E Testing

      From this list, select Choose Vue version, Router, and Vuex. This will allow you to choose your version of Vue and use Vuex and Vue Router.

      Next, choose 3.x (Preview) for your version of Vue, answer no (N) to history mode, and select the option to have your configurations In dedicated config file. Finally, answer N to avoid saving the setup for a future project.

      At this point, Vue will create your application.

      After the project creation, move into the folder using the command:

      To start, you’ll install Bulma, a free, open-source CSS framework based on Flexbox. Add Bulma to your project by running the following command:

      To use Bulma CSS in your project, open up your app’s entry point, the main.js file:

      Then add the following highlighted import line:

      vuex-shopping-cart/src/main.js

      import { createApp } from 'vue'
      import App from './App.vue'
      import router from './router'
      import store from './store'
      import './../node_modules/bulma/css/bulma.css'
      
      createApp(App).use(store).use(router).mount('#app')
      

      Save and close the file.

      In this app, you’ll use the Axios module to make requests to your server. Add the Axios module by running the following command:

      Now, run the app to make sure it is working:

      Navigate to http://localhost:8080 in your browser of choice. You will find the Vue app welcome page:

      Default Vue page when running your app in development mode

      Once you have confirmed that Vue is working, stop your server with CTRL+C.

      In this step, you globally installed Vue CLI in your computer, created a Vue project, installed the required npm packages Axios and Bulma, and imported Bulma to the project in the main.js file. Next, you will set up a back-end API to store data for your app.

      Step 2 — Setting Up the Backend

      In this step, you will create a separate backend to work with your Vue project. This will be in a different project folder from your front-end Vue application.

      First, move out of your Vue directory:

      Make a separate directory named cart-backend:

      Once you have your back-end folder, make it your working directory:

      You will get started by initializing the project with the necessary file. Create the file structure of your app with the following commands:

      • touch server.js
      • touch server-cart-data.json
      • touch server-product-data.json

      You use the touch command here to create empty files. The server.js file will hold your Node.js server, and the JSON will hold data for the shop’s products and the user’s shopping cart.

      Now run the following command to create a package.json file:

      For more information on npm and Node, check out our How To Code in Node.js series.

      Install these back-end dependencies into your Node project:

      • npm install concurrently express body-parser

      Express is a Node framework for web applications, which will provide useful abstractions for handling API requests. Concurrently will be used to run the Express back-end server and the Vue.js development server simulteneously. Finally, body-parser is an Express middleware that will parse requests to your API.

      Next, open a server.js file in the root of your application:

      Then add the following code:

      cart-backend/server.js

      const express = require('express');
      const bodyParser = require('body-parser');
      const fs = require('fs');
      const path = require('path');
      
      const app = express();
      const PRODUCT_DATA_FILE = path.join(__dirname, 'server-product-data.json');
      const CART_DATA_FILE = path.join(__dirname, 'server-cart-data.json');
      
      app.set('port', (process.env.PORT || 3000));
      app.use(bodyParser.json());
      app.use(bodyParser.urlencoded({ extended: true }));
      app.use((req, res, next) => {
        res.setHeader('Cache-Control', 'no-cache, no-store, must-revalidate');
        res.setHeader('Pragma', 'no-cache');
        res.setHeader('Expires', '0');
        next();
      });
      
      app.listen(app.get('port'), () => {
        console.log(`Find the server at: http://localhost:${app.get('port')}/`);
      });
      

      This snippet first adds the Node modules to your backend, including the fs module to write to your filesystem and the path module to make defining filepaths easier. You then initialize the Express app and save references to your JSON files as PRODUCT_DATA_FILE and CART_DATA_FILE. These will be used as data repositories. Finally, you created an Express server, set the port, created a middleware to set the response headers, and set the server to listen on your port. For more information on Express, see the official Express documentation.

      The setHeader method sets the header of the HTTP responses. In this case, you are using Cache-Control to direct the caching of your app. For more information on this, check out the Mozilla Developer Network article on Cache-Control.

      Next, you will create an API endpoint that your frontend will query to add an item to the shopping cart. To do this, you will use app.post to listen for an HTTP POST request.

      Add the following code to server.js just after the last app.use() middleware:

      cart-backend/server.js

      ...
      app.use((req, res, next) => {
        res.setHeader('Cache-Control', 'no-cache, no-store, must-revalidate');
        res.setHeader('Pragma', 'no-cache');
        res.setHeader('Expires', '0');
        next();
      });
      
      app.post('/cart', (req, res) => {
          fs.readFile(CART_DATA_FILE, (err, data) => {
            const cartProducts = JSON.parse(data);
            const newCartProduct = { 
              id: req.body.id,
              title: req.body.title,
              description: req.body.description,
              price: req.body.price,
              image_tag: req.body.image_tag, 
              quantity: 1 
            };
            let cartProductExists = false;
            cartProducts.map((cartProduct) => {
              if (cartProduct.id === newCartProduct.id) {
                cartProduct.quantity++;
                cartProductExists = true;
              }
            });
            if (!cartProductExists) cartProducts.push(newCartProduct);
            fs.writeFile(CART_DATA_FILE, JSON.stringify(cartProducts, null, 4), () => {
              res.setHeader('Cache-Control', 'no-cache');
              res.json(cartProducts);
            });
          });
        });
      
      app.listen(app.get('port'), () => {
        console.log(`Find the server at: http://localhost:${app.get('port')}/`);
      });
      

      This code receives the request object containing the cart items from the frontend and stores them in the server-cart-data.json file in the root of your project. Products here are JavaScript objects with id, title, description, price, image_tag, and quantity properties. The code also checks if the cart already exists to ensure that requests for a repeated product only increase the quantity.

      Now, add code to create an API endpoint to remove an item from the shopping cart. This time, you will use app.delete to listen for an HTTP DELETE request.

      Add the following code to server.js just after the previous endpoint:

      cart-backend/server.js

      ...
            fs.writeFile(CART_DATA_FILE, JSON.stringify(cartProducts, null, 4), () => {
              res.setHeader('Cache-Control', 'no-cache');
              res.json(cartProducts);
            });
          });
        });
      
      app.delete('/cart/delete', (req, res) => {
        fs.readFile(CART_DATA_FILE, (err, data) => {
          let cartProducts = JSON.parse(data);
          cartProducts.map((cartProduct) => {
            if (cartProduct.id === req.body.id && cartProduct.quantity > 1) {
              cartProduct.quantity--;
            } else if (cartProduct.id === req.body.id && cartProduct.quantity === 1) {
              const cartIndexToRemove = cartProducts.findIndex(cartProduct => cartProduct.id === req.body.id);
              cartProducts.splice(cartIndexToRemove, 1);
            }
          });
          fs.writeFile(CART_DATA_FILE, JSON.stringify(cartProducts, null, 4), () => {
            res.setHeader('Cache-Control', 'no-cache');
            res.json(cartProducts);
          });
        });
      });
      
      app.listen(app.get('port'), () => {
        console.log(`Find the server at: http://localhost:${app.get('port')}/`); // eslint-disable-line no-console
      });
      

      This code receives the request object containing the item to be removed from the cart and checks the server-cart-data.json file for this item via its id. If it exists and the quantity is greater than one, then the quantity of the item in the cart is deducted. Otherwise, if the item’s quantity is less than 1, it will be removed from the cart and the remaining items will be stored in the server-cart-data.json file.

      To give your user additional functionality, you can now create an API endpoint to remove all items from the shopping cart. This will also listen for a DELETE request.

      Add the following highlighted code to server.js after the previous endpoint:

      cart-backend/server.js

      ...
          fs.writeFile(CART_DATA_FILE, JSON.stringify(cartProducts, null, 4), () => {
            res.setHeader('Cache-Control', 'no-cache');
            res.json(cartProducts);
          });
        });
      });
      
      app.delete('/cart/delete/all', (req, res) => {
        fs.readFile(CART_DATA_FILE, () => {
          let emptyCart = [];
          fs.writeFile(CART_DATA_FILE, JSON.stringify(emptyCart, null, 4), () => {
            res.json(emptyCart);
          });
        });
      });
      
      app.listen(app.get('port'), () => {
        console.log(`Find the server at: http://localhost:${app.get('port')}/`); // eslint-disable-line no-console
      });
      

      This code is responsible for removing all the items from the cart by returning an empty array.

      Next, you will create an API endpoint to retrieve all the products from the product storage. This will use app.get to listen for a GET request.

      Add the following code to server.js after the previous endpoint:

      cart-backend/server.js

      ...
      app.delete('/cart/delete/all', (req, res) => {
        fs.readFile(CART_DATA_FILE, () => {
          let emptyCart = [];
          fs.writeFile(CART_DATA_FILE, JSON.stringify(emptyCart, null, 4), () => {
            res.json(emptyCart);
          });
        });
      });
      
      app.get("https://www.digitalocean.com/products", (req, res) => {
        fs.readFile(PRODUCT_DATA_FILE, (err, data) => {
          res.setHeader('Cache-Control', 'no-cache');
          res.json(JSON.parse(data));
        });
      });
      ...
      

      This code uses the file system’s native readFile method to fetch all the data in the server-product-data.json file and returns them in JSON format.

      Finally, you will create an API endpoint to retrieve all the items from the cart storage:

      cart-backend/server.js

      ...
      app.get("https://www.digitalocean.com/products", (req, res) => {
        fs.readFile(PRODUCT_DATA_FILE, (err, data) => {
          res.setHeader('Cache-Control', 'no-cache');
          res.json(JSON.parse(data));
        });
      });
      
      app.get('/cart', (req, res) => {
        fs.readFile(CART_DATA_FILE, (err, data) => {
          res.setHeader('Cache-Control', 'no-cache');
          res.json(JSON.parse(data));
        });
      });
      ...
      

      Similarly, this code uses the file system’s native readFile method to fetch all the data in the server-cart-data.json file and returns them in JSON format.

      Save and close the server.js file.

      Next, you will add some mock data to your JSON files for testing purposes.

      Open up the server-cart-data.json file you created earlier:

      • nano server-cart-data.json

      Add the following array of product objects:

      cart-backend/server-cart-data.json

      [
          {
              "id": 2,
              "title": "MIKANO Engine",
              "description": "Lorem ipsum dolor sit amet, consectetur  dignissimos suscipit voluptatibus distinctio, error nostrum expedita omnis ipsum sit inventore aliquam sunt quam quis! ",
              "price": 650.9,
              "image_tag": "diesel-engine.png",
              "quantity": 1
          },
          {
              "id": 3,
              "title": "SEFANG Engine",
              "description": "Lorem ipsum dolor sit amet, consectetur  dignissimos suscipit voluptatibus distinctio, error nostrum expedita omnis ipsum sit inventore aliquam sunt quam quis!",
              "price": 619.9,
              "image_tag": "sefang-engine.png",
              "quantity": 1
          }
      ]
      

      This shows two engines that will start out in the user’s shopping cart.

      Save and close the file.

      Now open the server-product-data.json file:

      • nano server-product-data.json

      Add the following data in server-product-data.json file:

      cart-backend/server-product-data.json

      [
          {
            "id": 1,
            "title": "CAT Engine",
            "description": "Lorem ipsum dolor sit amet, consectetur  dignissimos suscipit voluptatibus distinctio, error nostrum expedita omnis ipsum sit inventore aliquam sunt quam quis!",
            "product_type": "power set/diesel engine",
            "image_tag": "CAT-engine.png",
            "created_at": 2020,
            "owner": "Colton",
            "owner_photo": "image-colton.jpg",
            "email": "colt@gmail.com",
            "price": 719.9
          },
          {
            "id": 2,
            "title": "MIKANO Engine",
            "description": "Lorem ipsum dolor sit amet, consectetur  dignissimos suscipit voluptatibus distinctio, error nostrum expedita omnis ipsum sit inventore aliquam sunt quam quis! ",
            "product_type": "power set/diesel engine",
            "image_tag": "diesel-engine.png",
            "created_at": 2020,
            "owner": "Colton",
            "owner_photo": "image-colton.jpg",
            "email": "colt@gmail.com",
            "price": 650.9
          },
          {
            "id": 3,
            "title": "SEFANG Engine",
            "description": "Lorem ipsum dolor sit amet, consectetur  dignissimos suscipit voluptatibus distinctio, error nostrum expedita omnis ipsum sit inventore aliquam sunt quam quis!",
            "product_type": "power set/diesel engine",
            "image_tag": "sefang-engine.png",
            "created_at": 2017,
            "owner": "Anne",
            "owner_photo": "image-anne.jpg",
            "email": "anne@gmail.com",
            "price": 619.9
          },
          {
            "id": 4,
            "title": "CAT Engine",
            "description": "Lorem ipsum dolor sit amet, consectetur  dignissimos suscipit voluptatibus distinctio, error nostrum expedita omnis ipsum sit inventore aliquam sunt quam quis!",
            "product_type": "power set/diesel engine",
            "image_tag": "lawn-mower.png",
            "created_at": 2017,
            "owner": "Irene",
            "owner_photo": "image-irene.jpg",
            "email": "irene@gmail.com",
            "price": 319.9
          }
      
        ]
      

      This will hold all the possible products that the user can put in their cart.

      Save and close the file.

      Finally, execute this command to run the server:

      You will receive something like this on your terminal:

      Output

      Find the server at: http://localhost:3000/

      Leave this server running in this window.

      Finally, you will set up a proxy server in your Vue app. This will enable the connection between the frontend and backend.

      Go to the root directory of your Vue app:

      In the terminal, run this command to create a Vue configuration file:

      Then, add this code:

      vuex-shopping-cart/vue.config.js

      module.exports = {
        devServer: {
          proxy: {
            '/api': {
              target: 'http://localhost:3000/',
              changeOrigin: true,
              pathRewrite: {
                '^/api': ''
              }
            }
          }
        }
      }
      

      This will send requests from your frontend to your back-end server at http://localhost:3000/. For more information on proxy configuration, review the Vue devServer.proxy documentation.

      Save and close the file.

      In this step, you wrote server-side code that will handle API endpoints for your shopping cart. You started by creating the file structure and ended with adding necessary code in the server.js file and data in your JSON files. Next, you will set up the state storage for your frontend.

      Step 3 — Setting Up State Management with Vuex

      In Vuex, the store is where the state of the application is kept. The application state can only be updated by dispatching actions within a component that will then trigger mutations in the store. The Vuex store is made up of the state, mutations, actions, and getters.

      In this step, you’re going to build each of these pieces, after which you will couple everything together into a Vuex store.

      State

      Now you will create a place to store state for your application.

      The store folder in the root directory src of your project is automatically created at the time of the project setup. Locate the store folder in the src directory of your project then create a new folder named modules:

      Inside this folder, create the product and cart folders:

      • mkdir src/store/modules/product
      • mkdir src/store/modules/cart

      These will hold all the state files for your product inventory and your user’s cart. You will build these two files up at the same time, each open in a separate terminal. This way, you will be able to compare your mutations, getters, and actions side-by-side.

      Finally, open an index.js file in the product folder:

      • nano src/store/modules/product/index.js

      Add the following code to create a state object containing your productItems:

      vuex-shopping-cart/src/store/modules/product/index.js

      import axios from 'axios';
      const state = {
        productItems: [] 
      }
      

      Save the file and keep it open.

      Similarly, in a new terminal, add an index.js file to the cart directory with the following:

      • nano src/store/modules/cart/index.js

      Then add code for the cartItems:

      vuex-shopping-cart/src/store/modules/cart/index.js

      import axios from 'axios';
      const state = {
        cartItems: []
      }
      

      Save this file, but keep it open.

      In these code snippets, you imported the Axios module and set the state. The state is a store object that holds the application-level data that needs to be shared between components.

      Now that you’ve set the states, head over to mutations.

      Mutations

      Mutations are methods that modify the store state. They usually consist of a string type and a handler that accepts the state and payload as parameters.

      You will now create all the mutations for your application.

      Add the following code in the product/index.js file just after the state section:

      vuex-shopping-cart/src/store/modules/product/index.js

      ...
      const mutations = {
        UPDATE_PRODUCT_ITEMS (state, payload) {
          state.productItems = payload;
        }
      }
      

      This creates a mutations object that holds an UPDATE_PRODUCT_ITEMS method that sets the productItems array to the payload value.

      Similarly, add the following code in the cart/index.js file just after the state section:

      vuex-shopping-cart/src/store/modules/cart/index.js

      ...
      const mutations = {
        UPDATE_CART_ITEMS (state, payload) {
          state.cartItems = payload;
        }
      }
      

      This creates a similar UPDATE_CART_ITEMS for your user’s shopping cart. Note that this follows the Flux architecture style of making references to mutations in capital letters.

      Actions

      Actions are methods that will handle mutations, so that mutations are insulated from the rest of your application code.

      In product/index.js, create an actions object with all the actions for your application:

      vuex-shopping-cart/src/store/modules/product/index.js

      ...
      const actions = {
        getProductItems ({ commit }) {
          axios.get(`/api/products`).then((response) => {
            commit('UPDATE_PRODUCT_ITEMS', response.data)
          });
        }
      }
      

      Here the getProductItems method sends an asynchronous GET request to the server using the Axios package that you installed earlier. When the request is successful, the UPDATE_PRODUCT_ITEMS mutation is called with the response data as the payload.

      Next, add the following actions object to cart/index.js:

      vuex-shopping-cart/src/store/modules/cart/index.js

      ...
      const actions = {
        getCartItems ({ commit }) {
          axios.get('/api/cart').then((response) => {
            commit('UPDATE_CART_ITEMS', response.data)
          });
        },
        addCartItem ({ commit }, cartItem) {
          axios.post('/api/cart', cartItem).then((response) => {
            commit('UPDATE_CART_ITEMS', response.data)
          });
        },
        removeCartItem ({ commit }, cartItem) {
          axios.delete('/api/cart/delete', cartItem).then((response) => {
            commit('UPDATE_CART_ITEMS', response.data)
          });
        },
        removeAllCartItems ({ commit }) {
          axios.delete('/api/cart/delete/all').then((response) => {
            commit('UPDATE_CART_ITEMS', response.data)
          });
        }
      }
      
      

      In this file, you create the getCartItems method, which sends an asynchronous GET request to the server. When the request is successful, the UPDATE_CART_ITEMS mutation is called with the response data as the payload. The same happens with the removeAllCartItems method, although it makes a DELETE request to the server. The removeCartItem and addCartItem methods receives the cartItem object as a parameter for making a DELETE or POST request. After a successful request, the UPDATE_CART_ITEMS mutation is called with the response data as the payload.

      You used ES6 destructuring to decouple the commit method from the Vuex context object. This is similar to using context.commit.

      Getters

      Getters are to an application store what computed properties are to a component. They return computed information from store state methods that involve receiving computed state data.

      Next, create a getters object to get all the information for the product module:

      vuex-shopping-cart/src/store/modules/product/index.js

      ...
      const getters = {
        productItems: state => state.productItems,
        productItemById: (state) => (id) => {
          return state.productItems.find(productItem => productItem.id === id)
        }
      }
      

      Here, you made a method productItems that returns the list of product items in the state, followed by productItemById, a higher order function that returns a single product by its id.

      Next, create a getters object in cart/index.js:

      vuex-shopping-cart/src/store/modules/cart/index.js

      ...
      const getters = {
        cartItems: state => state.cartItems,
        cartTotal: state => {
          return state.cartItems.reduce((acc, cartItem) => {
            return (cartItem.quantity * cartItem.price) + acc;
          }, 0).toFixed(2);
        },
        cartQuantity: state => {
          return state.cartItems.reduce((acc, cartItem) => {
            return cartItem.quantity + acc;
          }, 0);
        }
      }
      

      In this snippet, you made the cartItems method, which returns the list of cart items in the state, followed by cartTotal, which returns the computed value of the total amount of cart items available for checkout. Finally, you made the cartQuantity method, which retuns the quantity of items in the cart.

      Exporting the Module

      The final part of the product and cart modules will export the state, mutations, actions, and getters objects so that other parts of the application can access them.

      In product/index.js, add the following code at the end of the file:

      vuex-shopping-cart/src/store/modules/product/index.js

      ...
      const productModule = {
        state,
        mutations,
        actions,
        getters
      }
      
      export default productModule;
      

      This collects all your state objects into the productModule object, then exports it as a module.

      Save product/index.js and close the file.

      Next, add similar code to cart/index.js:

      vuex-shopping-cart/src/store/modules/product/index.js

      ...
          const cartModule = {
        state,
        mutations,
        actions,
        getters
      }
      export default cartModule;
      

      This exports the module as cartModule.

      Setting up the Store

      With the state, mutations, actions, and getters all set up, the final part of integrating Vuex into your application is creating the store. Here you will harness the Vuex modules to split your application store into two manageable fragments.

      To create your store, open up the index.js file in your store folder:

      Add the following highlighted lines:

      vuex-shopping-cart/src/store/index.js

      import { createStore } from 'vuex'
      import product from'./modules/product';
      import cart from './modules/cart';
      
      export default createStore({
        modules: {
          product,
          cart
        }
      })
      

      Save the file, then exit the text editor.

      You have now created the methods needed for state management and have created the store for your shopping cart. Next you will create user interface (UI) components to consume the data.

      Step 4 — Creating Interface Components

      Now that you have the store for your shopping cart set up, you can move onto making the components for the user interface (UI). This will include making some changes to the router and making front-end components for your navigation bar and list and item views of your products and your cart.

      First, you will update your vue-router setup. Remember that when you used the Vue CLI tool to scaffold your application, you chose the router option, which allowed Vue to automatically set up the router for you. Now you can re-configure the router to provide paths for Cart_List.vue and Product_List.vue, which are Vue components you will make later.

      Open up the router file with the following command:

      • nano vuex-shopping-cart/src/router/index.js

      Add the following highlighted lines:

      vuex-shopping-cart/src/router/index.js

      import { createRouter, createWebHashHistory } from 'vue-router'
      import CartList from '../components/cart/Cart_List.vue';
      import ProductList from '../components/product/Product_List.vue';
      
      const routes = [
        {
          path: '/inventory',
          component: ProductList
        },
        {
          path: '/cart',
          component: CartList
        },
        {
          path: '/',
          redirect: '/inventory'
        },
      ]
      const router = createRouter({
        history: createWebHashHistory(),
        routes
      })
      
      export default router
      

      This creates the /inventory route for your products and the /cart route for the items in your cart. It also redirects your root path / to the product view.

      Once you have added this code, save and close the file.

      Now you can set up your UI component directories. Run this command on your terminal to move to the component’s directory:

      Run this command to create three new sub-folders under the component’s directory:

      core will hold essential parts of your application, such as the navigation bar. cart and product will hold the item and list views of the shopping cart and the total inventory.

      Under the core directory, create the Navbar.vue file by running this command:

      Under the cart directory, create the files Cart_List_Item.vue and Cart_List.vue:

      • touch cart/Cart_List_Item.vue cart/Cart_List.vue

      Finally, under the product directory, create these two files:

      • touch product/Product_List_Item.vue product/Product_List.vue

      Now that the file structure has been outlined, you can move on to creating the individual components of your front-end app.

      In the navbar, the cart navigation link will display the quantity of items in your cart. You will use the Vuex mapGetters helper method to directly map store getters with component computed properties, allowing your app to get this data from the store’s getters to the Navbar component.

      Open the navbar file:

      Replace the code with the following:

      vuex-shopping-cart/src/components/core/Navbar.vue

      <template>
          <nav class="navbar" role="navigation" aria-label="main navigation">
            <div class="navbar-brand">
              <a
                role="button"
                class="navbar-burger burger"
                aria-label="menu"
                aria-expanded="false"
                data-target="navbarBasicExample"
              >
                <span aria-hidden="true"></span>
                <span aria-hidden="true"></span>
                <span aria-hidden="true"></span>
              </a>
            </div>
            <div id="navbarBasicExample" class="navbar-menu">
              <div class="navbar-end">
                <div class="navbar-item">
                  <div class="buttons">
                    <router-link to="/inventory" class="button is-primary">
                     <strong> Inventory</strong>
                    </router-link>
                    <router-link to="/cart"  class="button is-warning">   <p>
          Total cart items:
          <span> {{cartQuantity}}</span> </p>
                    </router-link>
                  </div>
                </div>
              </div>
            </div>
          </nav>
      </template>
      <script>
      import {mapGetters} from "vuex"
      export default {
          name: "Navbar",
          computed: {
          ...mapGetters([
            'cartQuantity'
          ])
        },
        created() {
          this.$store.dispatch("getCartItems");
        }
      }
      </script>
      

      As a Vue component, this file starts out with a template element, which holds the HTML for the component. This snippet includes multiple navbar classes that use pre-made styles from the Bulma CSS framework. For more information, check out the Bulma documentation.

      This also uses the router-link elements to connect the app to your products and cart, and uses cartQuantity as a computed property to dynamically keep track of the number of items in your cart.

      The JavaScript is held in the script element, which also handles state management and exports the component. The getCartItems action gets dispatched when the navbar component is created, updating the store state with all the cart items from the response data received from the server. After this, the store getters recompute their return values and the cartQuantity gets rendered in the template. Without dispatching the getCartItems action on the created life cycle hook, the value of cartQuantity will be 0 until the store state is modified.

      Save and close the file.

      Product_List Component

      This component is the parent to the Product_List_Item component. It will be responsible for passing down the product items as props to the Product_List_Item (child) component.

      First, open up the file:

      • nano product/Product_List.vue

      Update Product_List.vue as follows:

      vuex-shopping-cart/src/components/product/Product_List.vue

      <template>
        <div class="container is-fluid">
          <div class="tile is-ancestor">
            <div class="tile is-parent" v-for="productItem in productItems" :key="productItem.id">
            <ProductListItem :productItem="productItem"/>
            </div>
          </div>
        </div>
      </template>
      <script>
      import { mapGetters } from 'vuex';
      import Product_List_Item from './Product_List_Item'
      export default {
        name: "ProductList",
        components: {
          ProductListItem:Product_List_Item
        },
        computed: {
          ...mapGetters([
            'productItems'
          ])
        },
        created() {
          this.$store.dispatch('getProductItems');
        }
      };
      </script>
      

      Similar to the Navbar component logic discussed earlier, here the Vuex mapGetters helper method directly maps store getters with component computed properties to get the productItems data from the store. The getProductItems action gets dispatched when the ProductList component is created, updating the store state with all the product items from the response data received from the server. After this, the store getters re-computes their return values and the productItems gets rendered in the template. Without dispatching the getProductItems action on the created life cycle hook, there will be no product item displayed in the template until the store state is modified.

      Product_List_Item Component

      This component will be the direct child component to the Product_List component. It will receive the productItem data as props from its parent and render them in the template.

      Open Product_List_Item.vue:

      • nano product/Product_List_Item.vue

      Then add the following code:

      vuex-shopping-cart/src/components/product/Product_List_Item.vue

      <template>
          <div class="card">
            <div class="card-content">
              <div class="content">
                <h4>{{ productItem.title }}</h4>
                <a
                  class="button is-rounded is-pulled-left"
                  @click="addCartItem(productItem)"
                >
                  <strong>Add to Cart</strong>
                </a>
                <br />
                <p class="mt-4">
                  {{ productItem.description }}
                </p>
              </div>
              <div class="media">
                <div class="media-content">
                  <p class="title is-6">{{ productItem.owner }}</p>
                  <p class="subtitle is-7">{{ productItem.email }}</p>
                </div>
                <div class="media-right">
                  <a class="button is-primary is-light">
                    <strong>$ {{ productItem.price }}</strong>
                  </a>
                </div>
              </div>
            </div>
          </div>
      </template>
      <script>
      import {mapActions} from 'vuex'
      export default {
        name: "ProductListItem",
        props: ["productItem"],
        methods: {
          ...mapActions(["addCartItem"]),
        },
      };
      </script>
      

      In addition to the mapGetters helper function used in the previous components, Vuex also provides you with the mapActions helper function to directly map the component method to the store’s actions. In this case, you use the mapAction helper function to map the component method to the addCartItem action in the store. Now you can add items to the cart.

      Save and close the file.

      Cart_List Component

      This component is responsible for displaying all the product items added to the cart and also the removal of all the items from the cart.

      To create this component, first open the file:

      Next, update Cart_List.vue as follows:

      vuex-shopping-cart/src/components/cart/Cart_List.vue

      <template>
        <div id="cart">
          <div class="cart--header has-text-centered">
            <i class="fa fa-2x fa-shopping-cart"></i>
          </div>
          <p v-if="!cartItems.length" class="cart-empty-text has-text-centered">
            Add some items to the cart!
          </p>
          <ul>
            <li class="cart-item" v-for="cartItem in cartItems" :key="cartItem.id">
                <CartListItem :cartItem="cartItem"/>
            </li>
            <div class="notification is-success">
              <button class="delete"></button>
              <p>
                Total Quantity:
                <span class="has-text-weight-bold">{{ cartQuantity }}</span>
              </p>
            </div>
            <br>
          </ul>
          <div class="buttons">
          <button :disabled="!cartItems.length" class="button is-info">
            Checkout (<span class="has-text-weight-bold">${{ cartTotal }}</span>)
          </button>
      
       <button class="button is-danger is-outlined" @click="removeAllCartItems">
          <span>Delete All items</span>
          <span class="icon is-small">
            <i class="fas fa-times"></i>
          </span>
        </button>
             </div>
        </div>
      </template>
      <script>
      import { mapGetters, mapActions } from "vuex";
      import CartListItem from "./Cart_List_Item";
      export default {
        name: "CartList",
        components: {
          CartListItem
        },
        computed: {
          ...mapGetters(["cartItems", "cartTotal", "cartQuantity"]),
        },
        created() {
          this.$store.dispatch("getCartItems");
        },
        methods: {
          ...mapActions(["removeAllCartItems"]),
        }
      };
      </script>
      

      This code uses a v-if statement in the template to conditionally render a message if the cart is empty. Otherwise, it iterates through the store of cart items and renders them to the page. You also loaded in the cartItems, cartTotal, and cartQuantity getters to compute the data properties, and brought in the removeAllCartItems action.

      Save and close the file.

      Cart_List_Item Component

      This component is the direct child component of the Cart_List component. It receives the cartItem data as props from its parent and renders them in the template. It is also responsible for incrementing and decrementing the quantity of items in the cart.

      Open up the file:

      • nano cart/Cart_List_Item.vue

      Update Cart_List_Item.vue as follows:

      vuex-shopping-cart/src/components/cart/Cart_List_Item.vue

      <template>
        <div class="box">
          <div class="cart-item__details">
            <p class="is-inline">{{cartItem.title}}</p>
            <div>
              <span class="cart-item--price has-text-info has-text-weight-bold">
                ${{cartItem.price}} X {{cartItem.quantity}}
              </span>
      
              <span>
                <i class="fa fa-arrow-circle-up cart-item__modify" @click="addCartItem(cartItem)"></i>
                <i class="fa fa-arrow-circle-down cart-item__modify" @click="removeCartItem(cartItem)"></i>
              </span>
            </div>
      
          </div>
        </div>
      </template>
      <script>
      import { mapActions } from 'vuex';
      export default {
        name: 'CartListItem',
        props: ['cartItem'],
        methods: {
          ...mapActions([
            'addCartItem',
            'removeCartItem'
          ])
        }
      }
      </script>
      

      Here, you are using the mapAction helper function to map the component method to the addCartItem and removeCartItem actions in the store.

      Save and close the file.

      Lastly, you will update the App.vue file to bring these components into your app. First, move back to the root folder of your project:

      Now open the file:

      Replace the contents with the following code:

      vuex-shopping-cart/src/App.vue

      <template>
        <div>
          <Navbar/>
          <div class="container mt-6">
            <div class="columns">
              <div class="column is-12 column--align-center">
                <router-view></router-view>
              </div>
            </div>
          </div>
        </div>
      </template>
      <script>
      import Navbar from './components/core/Navbar'
      export default {
        name: 'App',
        components: {
          Navbar
        }
      }
      </script>
      <style>
      html,
      body {
        height: 100%;
        background: #f2f6fa;
      }
      </style>
      

      App.vue is the root of your application defined in Vue component file format. Once you have made the changes, save and close the file.

      In this step, you set up the frontend of your shopping cart app by creating components for the navigation bar, the product inventory, and the shopping cart. You also used the store actions and getters that you created in a previous step. Next, you will get your application up and running.

      Step 5 — Running the Application

      Now that your app is ready, you can start the development server and try out the final product.

      Run the following command in the root of your front-end project:

      This will start a development server that allows you to view your app on http://localhost:8080. Also, make sure that your backend is running in a separate terminal; you can do this by running the following command in your cart-backend project:

      Once your backend and your frontend are running, navigate to http://localhost:8080 in your browser. You will find your functioning shopping cart application:

      Animation of user adding and deleting products from the shopping cart application

      Conclusion

      In this tutorial, you built an online shopping cart app using Vue.js and Vuex for data management. These techniques can be reused to form the basis of an e-commerce shopping application. If you would like to learn more about Vue.js, check out our Vue.js topic page.



      Source link

      How To Build a Discord Bot in Python on Ubuntu 20.04


      The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Discord is a popular voice and text messaging platform used by gamers, open-source communities, conference organizers, and more. It has gained popularity due to features like superb voice chat, various text channels, and extensibility using automated assistants or “bots.”

      In this guide, you will build a Discord bot using the Python programming language and deploy it to an Ubuntu 20.04 server. You’ll be using the Discord Python library, discord.py, to program your bot.

      Note: Discord has two different names for their chat/voice rooms. They refer to these as both Servers and Guilds, depending on where you are in the documentation. For this tutorial, we will use the term Guild. Just know that these terms can be used interchangeably and often are in the official Discord documentation.

      Prerequisites

      To complete this tutorial, you’ll need:

      Step 1 — Creating a Bot User For Your Discord Guild

      Before you start coding, you need to create and register your bot in the Discord developer portal.

      Sign in to the developer portal and click on the New Application button to start the process. A Discord application allows you to interact with the Discord API. Bots are bound to applications; that is how they function.

      Click the New Application Button

      Next, add a name for the application in the modal and press the Create button. For this tutorial, we’ll name ours SammySharkBot.

      Name Your Application

      Discord will create your application. Next, you’ll note a page with information regarding your bot. You can choose to add a picture to serve as an icon for your bot. Click on the Choose a Neat Icon button in the App Icon section of the page. This is not required.

      Add an optional icon for your bot

      If you did upload an icon, you’d note a dialog box prompting you to save your changes. Press Save Changes to save your icon.

      Save your changes

      Now that you’ve set up your application, it’s time to create a bot that is associated with the application. Navigate to the Bot link in the Settings navigation pane on the left-hand side of the screen.

      Add a bot to your application

      Now click the Add Bot button to add the bot to your application.

      Confirm the creation of the bot

      Once you’ve done that, a Bot will appear with the optional icon you set up earlier. Remember this page; you’ll need to come back to it later to retrieve the API Token.

      Bot confirmation page

      You’ll be prompted with a modal that states Adding a bot user gives your app visible life in Discord. However, this action is irrevocable! Choose wisely. Go ahead and press Yes, do it! to create the bot user.

      Next, navigate to the OAuth2 link in the Settings navigation pane on the left-hand side of the screen. You need to add the appropriate permissions so your bot can function properly.

      Navigate to OAuth2 menu

      You’ll need to add the scopes in which your bot can function. Only check the bot option because that’s all you want this bot to do. For more information about the other scopes, visit the Discord OAuth2 Documentation.

      Add bot designation

      Now scroll down and give your bot the following permissions: View Channels, Send Messages, Read Message History. This will give your bot a limited set of actions that it can perform. If you want to create a bot with no restrictions, you can select the Administrator option; this is not recommended.

      Once you have done this, a Discord URL followed by a Copy button will appear. Click that button to copy the link and open it in a new window in your browser.

      Add bot permissions and copy the link

      A prompt will appear for your bot to connect your Guild. You may have permission to install bots in multiple Guilds, so you may note more than one here. Select the Guild you wish to install your bot into and click Authorize.

      Add bot to your Guild

      Before a bot is added to your Guild, Discord shows you all the permissions the bot will have. This ensures that you are aware of what you are installing. If you don’t want the bot to have a specific permission, you can uncheck it. However, don’t do that for this bot because the bot won’t work otherwise. Click Authorize to accept and add the bot to your Guild.

      Authorize bot permissions

      Go back to your Guild and check the channel where posts about new members appear. You will note that your bot joined your Guild. Your bot will also appear in the member list on the right-hand side, although the bot will appear offline; this will change after you start the bot code.

      Check if your bot joined your guild

      Now that your bot is added to your Guild, you are ready to bring it to life with some Python code.

      Step 2 — Creating a Python Virtual Environment for Your Project

      Before you get started coding, you need to set up your Python developer environment. In this step, you will install and activate your Python requirements within a virtual environment for easier management.

      First, create a directory in your home directory that you can use to store all of your virtual environments:

      Now create your virtual environment using Python:

      • python3 -m venv ~/.venvs/discord

      This will create a directory called discord within your .venvs directory. Inside, it will install a local version of Python and a local version of pip. You can use this to install and configure an isolated Python environment for your project.

      Before you install your project’s Python requirements, activate the virtual environment:

      • source ~/.venvs/discord/bin/activate

      Your prompt should change to indicate that you are now operating within a Python virtual environment. It will look something like this: (discord)user@host:~$.

      With your virtual environment active, install discord.py with the local instance of pip:

      Note: Once you have activate your virtual environment (when your prompt has (discord) preceding it), use pip instead of pip3, even if you are using Python 3. The virtual environment’s copy of the tool is always named pip, regardless of the Python version.

      Now that you have the Discord package installed, you will need to save this requirement and its dependencies. This is good practice so you can recreate your developer environment as needed.

      Use pip to save your environment’s information to a requirements.txt file:

      • pip freeze > requirements.txt

      You now have the libraries necessary to build a discord bot.

      Step 3 — Building a Minimal Discord Bot

      You will now begin coding your Discord bot. Once completed, your bot will listen for certain phrases shared in a text chat, and then it will respond to them accordingly. Specifically, your bot will flip a coin on behalf of the users.

      In this step, you will build a minimal version of your bot. You will then add more functionality in Step 4 and Step 5.

      To begin, open a file named bot.py in your preferred text editor:

      Now add the following imports to the file: os, random, and discord. The os library will allow you to read valuable information, such as API Tokens and Guild Name from your environment variables. The random library will allow you to generate the output for your random events. And the discord library will provide you with the methods, objects, and decorators required to interact with the discord API. The code is as follows:

      bot.py

      import os
      import random
      import discord
      

      Next, you will need to retrieve the Discord API token and your bot’s guild from environment variables. You’ll use the method getenv from the os library in the Python standard library. Append the following code to bot.py:

      bot.py

      ...
      
      token = os.getenv("DISCORD_TOKEN")
      my_guild = os.getenv("DISCORD_GUILD")
      

      In version 1.5 of discord.py, Intents were introduced. This was a breaking change from prior versions of Discord libraries that allowed bots to subscribe to certain events that happen within the Guild. For this tutorial, you’ll set the client’s available intents to default, but you may need to revisit if you plan on interacting with the GUILD_PRESENCES or GUILD_MEMBERS intents. These are Privileged Gateway Intents.

      Append the following code to bot.py to properly set up the intents:

      bot.py

      ...
      
      intents = discord.Intents.default()
      client = discord.Client(intents=intents)
      

      Now, let’s write some code that responds to an event in Discord. Events are actions that happen on the Discord Guild, such as sending a message or joining a channel. For a full list of supported events, check the Discord Event Reference API.

      The first segment of code will activate on the on_ready event. This event triggers when your bot loads into a Guild. Note that the Discord API does not guarantee that this event only happens once. If you are planning on putting code here that should only run once, do some checks to ensure that it indeed only executes once.

      You’ll write a print statement that prints when your bot has successfully connected to your Guild. Discord bots require the use of async methods so that the bot sits in a ready state and waits for calls.

      Append the following code to the end of bot.py:

      bot.py

      ...
      
      @client.event
      async def on_ready():
          for guild in client.guilds:
              if guild.name == my_guild:
                  break
      
          print(
              f"{client.user} is connected to the following guild:n"
              f"{guild.name}(id: {guild.id})"
          )
      

      You use a for loop to find your Guild among all available Guilds, and then you print information about the Guild and your connection to it.

      Finally, you need to tell the client to run when the script is executed using your API token. Append the following line to bot.py to run your bot:

      bot.py

      ...
      
      client.run(token)
      

      At this point, your code should look like this:

      bot.py

      import os
      import random
      import discord
      
      token = os.getenv("DISCORD_TOKEN")
      my_guild = os.getenv("DISCORD_GUILD")
      
      intents = discord.Intents.default()
      client = discord.Client(intents=intents)
      
      @client.event
      async def on_ready():
          for guild in client.guilds:
              if guild.name == my_guild:
                  break
      
          print(
              f"{client.user} is connected to the following guild:n"
              f"{guild.name}(id: {guild.id})"
          )
      
      client.run(token)
      

      You have just written the skeleton code that will allow your bot to connect to a Guild. Now you will test your bot’s basic functionality.

      Step 4 — Testing Your Bot Locally

      Before you code any more, check that your bot can connect to Discord. Navigate back to the discord developer panel and click on the Bot link in the navigation bar on the left. From here, locate the Token underneath the bot username and click the Copy button to copy the token to your clipboard.

      Get Bot Token

      Once you have copied the token, return to the terminal with your activated virtual environment. Export the Discord token so it is available in the environment variables for your bot to read on startup:

      • export DISCORD_TOKEN=YOUR_DISCORD_TOKEN

      Now you need to get the Guild name that your bot will join. Navigate to your Discord Guild. The Guild name should be in the top left-hand corner of your Guild page. This was the name you chose when you created a Guild in the prerequisites section.

      Get Guild Name

      Now, export the Discord Guild so it is available in the environment variables for your bot to read on startup:

      • export DISCORD_GUILD=YOUR_DISCORD_GUILD

      Now that you have exported the necessary environment variables, you are ready to test your bot:

      Run the application:

      It might take a few seconds, but you will receive an output like this. Note that your bot number and id will differ:

      Output

      SammySharkBot#8143 is connected to the following Guild: SammyShark(id: 801529616116350998)

      Navigate back to your Discord Guild and you will note that your bot now has a green circle near its name. This indicates that the bot is online.

      Verify your bot is online

      You may notice in your terminal that your code does not terminate. This is because the bot is a constantly running process. To stop your bot at any time, press the CTRL + C key combo.

      Step 5 — Extending the Discord Bot to Perform Random Tasks

      Now that your bot is working, it’s time to extend its functionality. For your bot to respond to messages sent in the text chat, you’ll need to listen to the on_message event. This function takes in one argument, message, which contains the message object, including various attributes and methods for responding to messages.

      One thing that you’ll need to ensure is that your bot doesn’t answer itself. If the bot were to interpret its own message as a trigger, then it would create an infinite loop. To avoid this, check that the user sending the message isn’t the bot.

      Append the following code to bot.py to declare the on_message function and to check who sent the message:

      bot.py

      ...
      @client.event
      async def on_message(message):
          if message.author == client.user:
              return
      

      Lastly, add the code to flip the coin. You will also perform a string manipulation on the message content to lower case (or upper, if you prefer) so you don’t have to worry about matching exact case in strings. Use the randint() function in the Python random library to simulate flipping a coin. Finally, once you have crafted a message, you will send it to the channel using the method for sending a message. Remember, this is asynchronous programming, so you’ll have to use the await keyword when sending the message.

      Append the following code to bot.py within the on_message function; this will add the coin-flipping functionality:

      bot.py

          message_content = message.content.lower()
          if "flip a coin" in message_content:
              rand_int = random.randint(0, 1)
              if rand_int == 0:
                  results = "Heads"
              else:
                  results = "Tails"
              await message.channel.send(results)
      

      What follows is the finished code for the bot. Ensure that your code looks like this, or copy and paste this code:

      bot.py

      import os
      import random
      import discord
      
      token = os.getenv("DISCORD_TOKEN")
      my_guild = os.getenv("DISCORD_GUILD")
      
      intents = discord.Intents.default()
      client = discord.Client(intents=intents)
      
      
      @client.event
      async def on_ready():
          for guild in client.guilds:
              if guild.name == my_guild:
                  break
      
          print(
              f"{client.user} is connected to the following guild:n"
              f"{guild.name}(id: {guild.id})"
          )
      
      
      @client.event
      async def on_message(message):
          if message.author == client.user:
              return
      
          message_content = message.content.lower()
          if "flip a coin" in message_content:
              rand_int = random.randint(0, 1)
              if rand_int == 0:
                  results = "Heads"
              else:
                  results = "Tails"
              await message.channel.send(results)
      
      client.run(token)
      

      Now that the bot is finished, it’s time to test it. Save and close the file.

      Rerun the app:

      After the app connects to your Guild, navigate to the #general channel in your Discord Guild. Type flip a coin in the text chat, and the bot will respond with either Heads or Tails.

      Your bot should respond with Heads or Tails

      Now that you have confirmed that your bot is functional, you are ready to deploy it to production on an Ubuntu 20.04 server.

      Step 6 — Deploying to Ubuntu 20.04

      You can leave your bot running from your local machine, but having to leave a terminal active or worry about power outages can be annoying. For convenience, you can deploy it to a server.

      It is best practice to run your bot as a non-root user. This tutorial uses the username sammy.

      First, copy your code and Python library requirements to the server using the scp command. Send everything to your user’s root directory:

      • scp bot.py requirements.txt sammy@your_server_ip:~

      Next, ssh into your server:

      You will now enter commands into the server’s terminal.

      You need to install the proper Python system packages to run your bot on your server.

      Install python3, python3-venv, and screen:

      • sudo apt update && sudo apt install python3 python3-venv screen

      Now, you’re going to use a tool called screen to create a virtual terminal. Without this tool, if you were to exit your terminal while the bot was running, the process would terminate, and your bot would go offline. With this tool, you can connect and disconnect to sessions so your code remains running. To learn more about screen, check out our tutorial on installing and using screen.

      To start a screen session, use the following command:

      screen will prompt you with a license agreement. Press Return to continue.

      Now that you have a virtual session set up, you need to create a Python virtual environment to run your code.

      First, like before, create a directory to store your virtual environments:

      Then create a new virtual environment:

      • python3 -m venv ~/.venvs/discord

      Activate the virtual environment:

      • source ~/.venvs/discord/bin/activate

      Next, install the necessary libraries using pip:

      • pip install -r requirements.txt

      Before you can run your bot, you’ll need to export the DISCORD_TOKEN and DISCORD_GUILD so your bot can access the API Key and Guild information:

      • export DISCORD_TOKEN=YOUR_DISCORD_TOKEN
      • export DISCORD_GUILD=YOUR_DISCORD_GUILD

      Finally, run your bot:

      Your bot will start up and start accepting messages. You can disconnect from the screen session using the key combination CTRL + A + D. When you are ready to reconnect to the session, you can use this screen command:

      You have successfully built a Discord bot using Python. The application runs on a server and responds to certain phrases shared in a Guild.

      Conclusion

      In this tutorial, you set up a Discord Guild, built a Discord bot, installed the bot in your Guild, and deployed the bot to an Ubuntu 20.04 server. You’ve only begun to explore the Discord API. Bots are powerful and extendable. To explore other capabilities, check out the discord.py API docs.

      Alternately, if you wish learn more about Python programming, visit our tutorial series, How To Code in Python 3.



      Source link

      How To Build a Data Processing Pipeline Using Luigi in Python on Ubuntu 20.04


      The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Luigi is a Python package that manages long-running batch processing, which is the automated running of data processing jobs on batches of items. Luigi allows you to define a data processing job as a set of dependent tasks. For example, task B depends on the output of task A. And task D depends on the output of task B and task C. Luigi automatically works out what tasks it needs to run to complete a requested job.

      Overall Luigi provides a framework to develop and manage data processing pipelines. It was originally developed by Spotify, who use it to manage plumbing together collections of tasks that need to fetch and process data from a variety of sources. Within Luigi, developers at Spotify built functionality to help with their batch processing needs including handling of failures, the ability to automatically resolve dependencies between tasks, and visualization of task processing. Spotify uses Luigi to support batch processing jobs, including providing music recommendations to users, populating internal dashboards, and calculating lists of top songs.

      In this tutorial, you will build a data processing pipeline to analyze the most common words from the most popular books on Project Gutenburg. To do this, you will build a pipeline using the Luigi package. You will use Luigi tasks, targets, dependencies, and parameters to build your pipeline.

      Prerequisites

      To complete this tutorial, you will need the following:

      Step 1 — Installing Luigi

      In this step, you will create a clean sandbox environment for your Luigi installation.

      First, create a project directory. For this tutorial luigi-demo:

      Navigate into the newly created luigi-demo directory:

      Create a new virtual environment luigi-venv:

      • python3 -m venv luigi-venv

      And activate the newly created virtual environment:

      • . luigi-venv/bin/activate

      You will find (luigi-venv) appended to the front of your terminal prompt to indicate which virtual environment is active:

      Output

      (luigi-venv) username@hostname:~/luigi-demo$

      For this tutorial, you will need three libraries: luigi, beautifulsoup4, and requests. The requests library streamlines making HTTP requests; you will use it to download the Project Gutenberg book lists and the books to analyze. The beautifulsoup4 library provides functions to parse data from web pages; you will use it to parse out a list of the most popular books on the Project Gutenberg site.

      Run the following command to install these libraries using pip:

      • pip install wheel luigi beautifulsoup4 requests

      You will get a response confirming the installation of the latest versions of the libraries and all of their dependencies:

      Output

      Successfully installed beautifulsoup4-4.9.1 certifi-2020.6.20 chardet-3.0.4 docutils-0.16 idna-2.10 lockfile-0.12.2 luigi-3.0.1 python-daemon-2.2.4 python-dateutil-2.8.1 requests-2.24.0 six-1.15.0 soupsieve-2.0.1 tornado-5.1.1 urllib3-1.25.10

      You’ve installed the dependencies for your project. Now, you’ll move on to building your first Luigi task.

      Step 2 — Creating a Luigi Task

      In this step, you will create a “Hello World” Luigi task to demonstrate how they work.

      A Luigi task is where the execution of your pipeline and the definition of each task’s input and output dependencies take place. Tasks are the building blocks that you will create your pipeline from. You define them in a class, which contains:

      • A run() method that holds the logic for executing the task.
      • An output() method that returns the artifacts generated by the task. The run() method populates these artifacts.
      • An optional input() method that returns any additional tasks in your pipeline that are required to execute the current task. The run() method uses these to carry out the task.

      Create a new file hello-world.py:

      Now add the following code to your file:

      hello-world.py

      import luigi
      
      class HelloLuigi(luigi.Task):
      
          def output(self):
              return luigi.LocalTarget('hello-luigi.txt')
      
          def run(self):
              with self.output().open("w") as outfile:
                  outfile.write("Hello Luigi!")
      
      

      You define that HelloLuigi() is a Luigi task by adding the luigi.Task mixin to it.

      The output() method defines one or more Target outputs that your task produces. In the case of this example, you define a luigi.LocalTarget, which is a local file.

      Note: Luigi allows you to connect to a variety of common data sources including AWS S3 buckets, MongoDB databases, and SQL databases. You can find a complete list of supported data sources in the Luigi docs.

      The run() method contains the code you want to execute for your pipeline stage. For this example you are opening the output() target file in write mode, self.output().open("w") as outfile: and writing "Hello Luigi!" to it with outfile.write("Hello Luigi!").

      To execute the task you created, run the following command:

      • python -m luigi --module hello-world HelloLuigi --local-scheduler

      Here, you run the task using python -m instead of executing the luigi command directly; this is because Luigi can only execute code that is within the current PYTHONPATH. You can alternatively add PYTHONPATH='.' to the front of your Luigi command, like so:

      • PYTHONPATH='.' luigi --module hello-world HelloLuigi --local-scheduler

      With the --module hello-world HelloLuigi flag, you tell Luigi which Python module and Luigi task to execute.

      The --local-scheduler flag tells Luigi to not connect to a Luigi scheduler and, instead, execute this task locally. (We explain the Luigi scheduler in Step 4.) Running tasks using the local-scheduler flag is only recommended for development work.

      Luigi will output a summary of the executed tasks:

      Output

      ===== Luigi Execution Summary ===== Scheduled 1 tasks of which: * 1 ran successfully: - 1 HelloLuigi() This progress looks :) because there were no failed tasks or missing dependencies ===== Luigi Execution Summary =====

      And it will create a new file hello-luigi.txt with content:

      hello-luigi.txt

      Hello Luigi!
      

      You have created a Luigi task that generates a file and then executed it using the Luigi local-scheduler. Now, you’ll create a task that can extract a list of books from a web page.

      In this step, you will create a Luigi task and define a run() method for the task to download a list of the most popular books on Project Gutenberg. You’ll define an output() method to store links to these books in a file. You will run these using the Luigi local scheduler.

      Create a new directory data inside of your luigi-demo directory. This will be where you will store the files defined in the output() methods of your tasks. You need to create the directories before running your tasks—Python throws exceptions when you try to write a file to a directory that does not exist yet:

      • mkdir data
      • mkdir data/counts
      • mkdir data/downloads

      Create a new file word-frequency.py:

      Insert the following code, which is a Luigi task to extract a list of links to the top most-read books on Project Gutenberg:

      word-frequency.py

      import requests
      import luigi
      from bs4 import BeautifulSoup
      
      
      class GetTopBooks(luigi.Task):
          """
          Get list of the most popular books from Project Gutenberg
          """
      
          def output(self):
              return luigi.LocalTarget("data/books_list.txt")
      
          def run(self):
              resp = requests.get("http://www.gutenberg.org/browse/scores/top")
      
              soup = BeautifulSoup(resp.content, "html.parser")
      
              pageHeader = soup.find_all("h2", string="Top 100 EBooks yesterday")[0]
              listTop = pageHeader.find_next_sibling("ol")
      
              with self.output().open("w") as f:
                  for result in listTop.select("li>a"):
                      if "/ebooks/" in result["href"]:
                          f.write("http://www.gutenberg.org{link}.txt.utf-8n"
                              .format(
                                  link=result["href"]
                              )
                          )
      

      You define an output() target of file "data/books_list.txt" to store the list of books.

      In the run() method, you:

      • use the requests library to download the HTML contents of the Project Gutenberg top books page.
      • use the BeautifulSoup library to parse the contents of the page. The BeautifulSoup library allows us to scrape information out of web pages. To find out more about using the BeautifulSoup library, read the How To Scrape Web Pages with Beautiful Soup and Python 3 tutorial.
      • open the output file defined in the output() method.
      • iterate over the HTML structure to get all of the links in the Top 100 EBooks yesterday list. For this page, this is locating all links <a> that are within a list item <li>. For each of those links, if they link to a page that points at a link containing /ebooks/, you can assume it is a book and write that link to your output() file.

      Screenshot of the Project Gutenberg top books web page with the top ebooks links highlighted

      Save and exit the file once you’re done.

      Execute this new task using the following command:

      • python -m luigi --module word-frequency GetTopBooks --local-scheduler

      Luigi will output a summary of the executed tasks:

      Output

      ===== Luigi Execution Summary ===== Scheduled 1 tasks of which: * 1 ran successfully: - 1 GetTopBooks() This progress looks :) because there were no failed tasks or missing dependencies ===== Luigi Execution Summary =====

      In the data directory, Luigi will create a new file (data/books_list.txt). Run the following command to output the contents of the file:

      This file contains a list of URLs extracted from the Project Gutenberg top projects list:

      Output

      http://www.gutenberg.org/ebooks/1342.txt.utf-8 http://www.gutenberg.org/ebooks/11.txt.utf-8 http://www.gutenberg.org/ebooks/2701.txt.utf-8 http://www.gutenberg.org/ebooks/1661.txt.utf-8 http://www.gutenberg.org/ebooks/16328.txt.utf-8 http://www.gutenberg.org/ebooks/45858.txt.utf-8 http://www.gutenberg.org/ebooks/98.txt.utf-8 http://www.gutenberg.org/ebooks/84.txt.utf-8 http://www.gutenberg.org/ebooks/5200.txt.utf-8 http://www.gutenberg.org/ebooks/51461.txt.utf-8 ...

      You’ve created a task that can extract a list of books from a web page. In the next step, you’ll set up a central Luigi scheduler.

      Step 4 — Running the Luigi Scheduler

      Now, you’ll launch the Luigi scheduler to execute and visualize your tasks. You will take the task developed in Step 3 and run it using the Luigi scheduler.

      So far, you have been running Luigi using the --local-scheduler tag to run your jobs locally without allocating work to a central scheduler. This is useful for development, but for production usage it is recommended to use the Luigi scheduler. The Luigi scheduler provides:

      • A central point to execute your tasks.
      • Visualization of the execution of your tasks.

      To access the Luigi scheduler interface, you need to enable access to port 8082. To do this, run the following command:

      To run the scheduler execute the following command:

      • sudo sh -c ". luigi-venv/bin/activate ;luigid --background --port 8082"

      Note: We have re-run the virtualenv activate script as root, before launching the Luigi scheduler as a background task. This is because when running sudo the virtualenv environment variables and aliases are not carried over.

      If you do not want to run as root, you can run the Luigi scheduler as a background process for the current user. This command runs the Luigi scheduler in the background and hides messages from the scheduler background task. You can find out more about managing background processes in the terminal at How To Use Bash’s Job Control to Manage Foreground and Background Processes:

      • luigid --port 8082 > /dev/null 2> /dev/null &

      Open a browser to access the Luigi interface. This will either be at http://your_server_ip:8082, or if you have set up a domain for your server http://your_domain:8082. This will open the Luigi user interface.

      Luigi default user interface

      By default, Luigi tasks run using the Luigi scheduler. To run one of your previous tasks using the Luigi scheduler omit the --local-scheduler argument from the command. Re-run the task from Step 3 using the following command:

      • python -m luigi --module word-frequency GetTopBooks

      Refresh the Luigi scheduler user interface. You will find the GetTopBooks task added to the run list and its execution status.

      Luigi User Interface after running the GetTopBooks Task

      You will continue to refer back to this user interface to monitor the progress of your pipeline.

      Note: If you’d like to secure your Luigi scheduler through HTTPS, you can serve it through Nginx. To set up an Nginx server using HTTPS follow: How To Secure Nginx with Let’s Encrypt on Ubuntu 20.04. See Github - Luigi - Pull Request 2785 for suggestions on a suitable Nginx configuration to connect the Luigi server to Nginx.

      You’ve launched the Luigi Scheduler and used it to visualize your executed tasks. Next, you will create a task to download the list of books that the GetTopBooks() task outputs.

      Step 5 — Downloading the Books

      In this step you will create a Luigi task to download a specified book. You will define a dependency between this newly created task and the task created in Step 3.

      First open your file:

      Add an additional class following your GetTopBooks() task to the word-frequency.py file with the following code:

      word-frequency.py

      . . .
      class DownloadBooks(luigi.Task):
          """
          Download a specified list of books
          """
          FileID = luigi.IntParameter()
      
          REPLACE_LIST = """.,"';_[]:*-"""
      
          def requires(self):
              return GetTopBooks()
      
          def output(self):
              return luigi.LocalTarget("data/downloads/{}.txt".format(self.FileID))
      
          def run(self):
              with self.input().open("r") as i:
                  URL = i.read().splitlines()[self.FileID]
      
                  with self.output().open("w") as outfile:
                      book_downloads = requests.get(URL)
                      book_text = book_downloads.text
      
                      for char in self.REPLACE_LIST:
                          book_text = book_text.replace(char, " ")
      
                      book_text = book_text.lower()
                      outfile.write(book_text)
      

      In this task you introduce a Parameter; in this case, an integer parameter. Luigi parameters are inputs to your tasks that affect the execution of the pipeline. Here you introduce a parameter FileID to specify a line in your list of URLs to fetch.

      You have added an additional method to your Luigi task, def requires(); in this method you define the Luigi task that you need the output of before you can execute this task. You require the output of the GetTopBooks() task you defined in Step 3.

      In the output() method, you define your target. You use the FileID parameter to create a name for the file created by this step. In this case, you format data/downloads/{FileID}.txt.

      In the run() method, you:

      • open the list of books generated in the GetTopBooks() task.
      • get the URL from the line specified by parameter FileID.
      • use the requests library to download the contents of the book from the URL.
      • filter out any special characters inside the book like :,.?, so they don’t get included in your word analysis.
      • convert the text to lowercase so you can compare words with different cases.
      • write the filtered output to the file specified in the output() method.

      Save and exit your file.

      Run the new DownloadBooks() task using this command:

      • python -m luigi --module word-frequency DownloadBooks --FileID 2

      In this command, you set the FileID parameter using the --FileID argument.

      Note: Be careful when defining a parameter with an _ in the name. To reference them in Luigi you need to substitute the _ for a -. For example, a File_ID parameter would be referenced as --File-ID when calling a task from the terminal.

      You will receive the following output:

      Output

      ===== Luigi Execution Summary ===== Scheduled 2 tasks of which: * 1 complete ones were encountered: - 1 GetTopBooks() * 1 ran successfully: - 1 DownloadBooks(FileID=2) This progress looks :) because there were no failed tasks or missing dependencies ===== Luigi Execution Summary =====

      Note from the output that Luigi has detected that you have already generated the output of GetTopBooks() and skipped running that task. This functionality allows you to minimize the number of tasks you have to execute as you can re-use successful output from previous runs.

      You have created a task that uses the output of another task and downloads a set of books to analyze. In the next step, you will create a task to count the most common words in a downloaded book.

      Step 6 — Counting Words and Summarizing Results

      In this step, you will create a Luigi task to count the frequency of words in each of the books downloaded in Step 5. This will be your first task that executes in parallel.

      First open your file again:

      Add the following imports to the top of word-frequency.py:

      word-frequency.py

      from collections import Counter
      import pickle
      

      Add the following task to word-frequency.py, after your DownloadBooks() task. This task takes the output of the previous DownloadBooks() task for a specified book, and returns the most common words in that book:

      word-frequency.py

      class CountWords(luigi.Task):
          """
          Count the frequency of the most common words from a file
          """
      
          FileID = luigi.IntParameter()
      
          def requires(self):
              return DownloadBooks(FileID=self.FileID)
      
          def output(self):
              return luigi.LocalTarget(
                  "data/counts/count_{}.pickle".format(self.FileID),
                  format=luigi.format.Nop
              )
      
          def run(self):
              with self.input().open("r") as i:
                  word_count = Counter(i.read().split())
      
                  with self.output().open("w") as outfile:
                      pickle.dump(word_count, outfile)
      

      When you define requires() you pass the FileID parameter to the next task. When you specify that a task depends on another task, you specify the parameters you need the dependent task to be executed with.

      In the run() method you:

      • open the file generated by the DownloadBooks() task.
      • use the built-in Counter object in the collections library. This provides an easy way to analyze the most common words in a book.
      • use the pickle library to store the output of the Python Counter object, so you can re-use that object in a later task. pickle is a library that you use to convert Python objects into a byte stream, which you can store and restore into a later Python session. You have to set the format property of the luigi.LocalTarget to allow it to write the binary output the pickle library generates.

      Save and exit your file.

      Run the new CountWords() task using this command:

      • python -m luigi --module word-frequency CountWords --FileID 2

      Open the CountWords task graph view in the Luigi scheduler user interface.

      Showing how to view a graph from the Luigi user interface

      Deselect the Hide Done option, and deselect Upstream Dependencies. You will find the flow of execution from the tasks you have created.

      Visualizing the execution of the CountWords task

      You have created a task to count the most common words in a downloaded book and visualized the dependencies between those tasks. Next, you will define parameters that you can use to customize the execution of your tasks.

      Step 7 — Defining Configuration Parameters

      In this step, you will add configuration parameters to the pipeline. These will allow you to customize how many books to analyze and the number of words to include in the results.

      When you want to set parameters that are shared among tasks, you can create a Config() class. Other pipeline stages can reference the parameters defined in the Config() class; these are set by the pipeline when executing a job.

      Add the following Config() class to the end of word-frequency.py. This will define two new parameters in your pipeline for the number of books to analyze and the number of most frequent words to include in the summary:

      word-frequency.py

      class GlobalParams(luigi.Config):
          NumberBooks = luigi.IntParameter(default=10)
          NumberTopWords = luigi.IntParameter(default=500)
      

      Add the following class to word-frequency.py. This class aggregates the results from all of the CountWords() task to create a summary of the most frequent words:

      word-frequency.py

      class TopWords(luigi.Task):
          """
          Aggregate the count results from the different files
          """
      
          def requires(self):
              requiredInputs = []
              for i in range(GlobalParams().NumberBooks):
                  requiredInputs.append(CountWords(FileID=i))
              return requiredInputs
      
          def output(self):
              return luigi.LocalTarget("data/summary.txt")
      
          def run(self):
              total_count = Counter()
              for input in self.input():
                  with input.open("rb") as infile:
                      nextCounter = pickle.load(infile)
                      total_count += nextCounter
      
              with self.output().open("w") as f:
                  for item in total_count.most_common(GlobalParams().NumberTopWords):
                      f.write("{0: <15}{1}n".format(*item))
      
      

      In the requires() method, you can provide a list where you want a task to use the output of multiple dependent tasks. You use the GlobalParams().NumberBooks parameter to set the number of books you need word counts from.

      In the output() method, you define a data/summary.txt output file that will be the final output of your pipeline.

      In the run() method you:

      • create a Counter() object to store the total count.
      • open the file and “unpickle” it (convert it from a file back to a Python object), for each count carried out in the CountWords() method
      • append the loaded count and add it to the total count.
      • write the most common words to target output file.

      Run the pipeline with the following command:

      • python -m luigi --module word-frequency TopWords --GlobalParams-NumberBooks 15 --GlobalParams-NumberTopWords 750

      Luigi will execute the remaining tasks needed to generate the summary of the top words:

      Output

      ===== Luigi Execution Summary ===== Scheduled 31 tasks of which: * 2 complete ones were encountered: - 1 CountWords(FileID=2) - 1 GetTopBooks() * 29 ran successfully: - 14 CountWords(FileID=0,1,10,11,12,13,14,3,4,5,6,7,8,9) - 14 DownloadBooks(FileID=0,1,10,11,12,13,14,3,4,5,6,7,8,9) - 1 TopWords() This progress looks :) because there were no failed tasks or missing dependencies ===== Luigi Execution Summary =====

      You can visualize the execution of the pipeline from the Luigi scheduler. Select the GetTopBooks task in the task list and press the View Graph button.

      Showing how to view a graph from the Luigi user interface

      Deselect the Hide Done and Upstream Dependencies options.

      Visualizing the execution of the TopWords Task

      It will show the flow of processing that is happening in Luigi.

      Open the data/summary.txt file:

      You will find the calculated most common words:

      Output

      the 64593 and 41650 of 31896 to 31368 a 25265 i 23449 in 19496 it 16282 that 15907 he 14974 ...

      In this step, you have defined and used parameters to customize the execution of your tasks. You have generated a summary of the most common words for a set of books.

      Find all the code for this tutorial in this repository.

      Conclusion

      This tutorial has introduced you to using the Luigi data processing pipeline and its major features including tasks, parameters, configuration parameters, and the Luigi scheduler.

      Luigi supports connecting to a large number of common data sources out the box. You can also scale it to run large, complex data pipelines. This provides a powerful framework to start solving your data processing challenges.

      For more tutorials, check out our Data Analysis topic page and Python topic page.



      Source link