One place for hosting & domains

      How To Build and Deploy a GraphQL Server with Node.js and MongoDB on Ubuntu 18.04


      The author selected the Wikimedia Foundation to receive a donation as part of the Write for DOnations program.

      Introduction

      GraphQL was publicly released by Facebook in 2015 as a query language for APIs that makes it easy to query and mutate data from different data collections. From a single endpoint, you can query and mutate multiple data sources with a single POST request. GraphQL solves some of the common design flaws in REST API architectures, such as situations where the endpoint returns more information than you actually need. Also, it is possible when using REST APIs you would need to send requests to multiple REST endpoints to collect all the information you require—a situation that is called the n+1 problem. An example of this would be when you want to show a users’ information, but need to collect data such as personal details and addresses from different endpoints.

      These problems don’t apply to GraphQL as it has only one endpoint, which can return data from multiple collections. The data it returns depends on the query that you send to this endpoint. In this query you define the structure of the data you want to receive, including any nested data collections. In addition to a query, you can also use a mutation to change data on a GraphQL server, and a subscription to watch for changes in the data. For more information about GraphQL and its concepts, you can visit the documentation on the official website.

      As GraphQL is a query language with a lot of flexibility, it combines especially well with document-based databases like MongoDB. Both technologies are based on hierarchical, typed schemas and are popular within the JavaScript community. Also, MongoDB’s data is stored as JSON objects, so no additional parsing is necessary on the GraphQL server.

      In this tutorial, you’ll build and deploy a GraphQL server with Node.js that can query and mutate data from a MongoDB database that is running on Ubuntu 18.04. At the end of this tutorial, you’ll be able to access data in your database by using a single endpoint, both by sending requests to the server directly through the terminal and by using the pre-made GraphiQL playground interface. With this playground you can explore the contents of the GraphQL server by sending queries, mutations, and subscriptions. Also, you can find visual representations of the schemas that are defined for this server.

      At the end of this tutorial, you’ll use the GraphiQL playground to quickly interface with your GraphQL server:

      The GraphiQL playground in action

      Prerequisites

      Before you begin this guide you’ll need the following:

      Step 1 — Setting Up the MongoDB Database

      Before creating the GraphQL server, make sure your database is configured right, has authentication enabled, and is filled with sample data. For this you need to connect to the Ubuntu 18.04 server running the MongoDB database from your command prompt. All steps in this tutorial will take place on this server.

      After you’ve established the connection, run the following command to check if MongoDB is active and running on your server:

      • sudo systemctl status mongodb

      You’ll see the following output in your terminal, indicating the MongoDB database is actively running:

      Output

      ● mongodb.service - An object/document-oriented database Loaded: loaded (/lib/systemd/system/mongodb.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2019-02-23 12:23:03 UTC; 1 months 13 days ago Docs: man:mongod(1) Main PID: 2388 (mongod) Tasks: 25 (limit: 1152) CGroup: /system.slice/mongodb.service └─2388 /usr/bin/mongod --unixSocketPrefix=/run/mongodb --config /etc/mongodb.conf

      Before creating the database where you’ll store the sample data, you need to create an admin user first, since regular users are scoped to a specific database. You can do this by executing the following command that opens the MongoDB shell:

      With the MongoDB shell you'll get direct access to the MongoDB database and can create users or databases and query data. Inside this shell, execute the following command that will add a new admin user to MongoDB. You can replace the highlighted keywords with your own username and password combination, but don't forget to write them down somewhere.

      • use admin
      • db.createUser({
      • user: "admin_username",
      • pwd: "admin_password",
      • roles: [{ role: "root", db: "admin"}]
      • })

      The first line of the preceding command selects the database called admin, which is the database where all the admin roles are stored. With the method db.createUser() you can create the actual user and define its username, password, and roles.

      Executing this command will return:

      Output

      Successfully added user: { "user" : "admin_username", "roles" : [ { "role" : "root", "db" : "admin" } ] }

      You can now close the MongoDB shell by typing exit.

      Next, log in at the MongoDB shell again, but this time with the newly created admin user:

      • mongo -u "admin_username" -p "admin_password" --authenticationDatabase "admin"

      This command will open the MongoDB shell as a specific user, where the -u flag specifies the username and the -p flag the password of that user. The extra flag --authenticationDatabase specifies that you want to log in as an admin.

      Next, you'll switch to a new database and then use the db.createUser() method to create a new user with permissions to make changes to this database. Replace the highlighted sections with your own information, making sure to write these credentials down.

      Run the following command in the MongoDB shell:

      • use database_name
      • db.createUser({
      • user: "username",
      • pwd: "password",
      • roles: ["readWrite"]
      • })

      This will return the following:

      Output

      Successfully added user: { "user" : "username", "roles" : ["readWrite"] }

      After creating the database and user, fill this database with sample data that can be queried by the GraphQL server later on in this tutorial. For this, you can use the bios collection sample from the MongoDB website. By executing the commands in the following code snippet you'll insert a smaller version of this bios collection dataset into your database. You can replace the highlighted sections with your own information, but for the purposes of this tutorial, name the collection bios:

      • db.bios.insertMany([
      • {
      • "_id" : 1,
      • "name" : {
      • "first" : "John",
      • "last" : "Backus"
      • },
      • "birth" : ISODate("1924-12-03T05:00:00Z"),
      • "death" : ISODate("2007-03-17T04:00:00Z"),
      • "contribs" : [
      • "Fortran",
      • "ALGOL",
      • "Backus-Naur Form",
      • "FP"
      • ],
      • "awards" : [
      • {
      • "award" : "W.W. McDowell Award",
      • "year" : 1967,
      • "by" : "IEEE Computer Society"
      • },
      • {
      • "award" : "National Medal of Science",
      • "year" : 1975,
      • "by" : "National Science Foundation"
      • },
      • {
      • "award" : "Turing Award",
      • "year" : 1977,
      • "by" : "ACM"
      • },
      • {
      • "award" : "Draper Prize",
      • "year" : 1993,
      • "by" : "National Academy of Engineering"
      • }
      • ]
      • },
      • {
      • "_id" : ObjectId("51df07b094c6acd67e492f41"),
      • "name" : {
      • "first" : "John",
      • "last" : "McCarthy"
      • },
      • "birth" : ISODate("1927-09-04T04:00:00Z"),
      • "death" : ISODate("2011-12-24T05:00:00Z"),
      • "contribs" : [
      • "Lisp",
      • "Artificial Intelligence",
      • "ALGOL"
      • ],
      • "awards" : [
      • {
      • "award" : "Turing Award",
      • "year" : 1971,
      • "by" : "ACM"
      • },
      • {
      • "award" : "Kyoto Prize",
      • "year" : 1988,
      • "by" : "Inamori Foundation"
      • },
      • {
      • "award" : "National Medal of Science",
      • "year" : 1990,
      • "by" : "National Science Foundation"
      • }
      • ]
      • }
      • ]);

      This code block is an array consisting of multiple objects that contain information about successful scientists from the past. After running these commands to enter this collection into your database, you'll receive the following message indicating the data was added:

      Output

      { "acknowledged" : true, "insertedIds" : [ 1, ObjectId("51df07b094c6acd67e492f41") ] }

      After seeing the success message, you can close the MongoDB shell by typing exit. Next, configure the MongoDB installation to have authorization enabled so only authenticated users can access the data. To edit the configuration of the MongoDB installation, open the file containing the settings for this installation:

      • sudo nano /etc/mongodb.conf

      Uncomment the highlighted line in the following code to enable authorization:

      /etc/mongodb.conf

      ...
      # Turn on/off security.  Off is currently the default
      #noauth = true
      auth = true
      ...
      

      In order to make these changes active, restart MongoDB by running:

      • sudo systemctl restart mongodb

      Make sure the database is running again by executing the command:

      • sudo systemctl status mongodb

      This will yield output similar to the following:

      Output

      ● mongodb.service - An object/document-oriented database Loaded: loaded (/lib/systemd/system/mongodb.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2019-02-23 12:23:03 UTC; 1 months 13 days ago Docs: man:mongod(1) Main PID: 2388 (mongod) Tasks: 25 (limit: 1152) CGroup: /system.slice/mongodb.service └─2388 /usr/bin/mongod --unixSocketPrefix=/run/mongodb --config /etc/mongodb.conf

      To make sure that your user can connect to the database you just created, try opening the MongoDB shell as an authenticated user with the command:

      • mongo -u "username" -p "password" --authenticationDatabase "database_name"

      This uses the same flags as before, only this time the --authenticationDatabase is set to the database you've created and filled with the sample data.

      Now you've successfully added an admin user and another user that has read/write access to the database with the sample data. Also, the database has authorization enabled meaning you need a username and password to access it. In the next step you'll create the GraphQL server that will be connected to this database later in the tutorial.

      Step 2 — Creating the GraphQL Server

      With the database configured and filled with sample data, it's time to create a GraphQL server that can query and mutate this data. For this you'll use Express and express-graphql, which both run on Node.js. Express is a lightweight framework to quickly create Node.js HTTP servers, and express-graphql provides middleware to make it possible to quickly build GraphQL servers.

      The first step is to make sure your machine is up to date:

      Next, install Node.js on your server by running the following commands. Together with Node.js you'll also install npm, a package manager for JavaScript that runs on Node.js.

      • sudo apt install nodejs npm

      After following the installation process, check if the Node.js version you've just installed is v8.10.0 or higher:

      This will return the following:

      Output

      v8.10.0

      To initialize a new JavaScript project, run the following commands on the server as a sudo user, and replace the highlighted keywords with a name for your project.

      First move into the root directory of your server:

      Once there, create a new directory named after your project:

      Move into this directory:

      Finally, initialize a new npm package with the following command:

      After running npm init -y you'll receive a success message that the following package.json file was created:

      Output

      Wrote to /home/username/project_name/package.json: { "name": "project_name", "version": "1.0.0", "description": "", "main": "index.js", "scripts": { "test": "echo "Error: no test specified" && exit 1" }, "keywords": [], "author": "", "license": "ISC" }

      Note: You can also execute npm init without the -y flag, after which you would answer multiple questions to set up the project name, author, etc. You can enter the details or just press enter to proceed.

      Now that you've initialized the project, install the packages you need to set up the GraphQL server:

      • sudo npm install --save express express-graphql graphql

      Create a new file called index.js and subsequently open this file by running:

      Next, add the following code block into the newly created file to set up the GraphQL server:

      index.js

      const express = require('express');
      const graphqlHTTP = require('express-graphql');
      const { buildSchema } = require('graphql');
      
      // Construct a schema, using GraphQL schema language
      const schema = buildSchema(`
        type Query {
          hello: String
        }
      `);
      
      // Provide resolver functions for your schema fields
      const resolvers = {
        hello: () => 'Hello world!'
      };
      
      const app = express();
      app.use('/graphql', graphqlHTTP({
        schema,
        rootValue: resolvers
      }));
      app.listen(4000);
      
      console.log(`🚀 Server ready at http://localhost:4000/graphql`);
      

      This code block consists of several parts that are all important. First you describe the schema of the data that is returned by the GraphQL API:

      index.js

      ...
      // Construct a schema, using GraphQL schema language
      const schema = buildSchema(`
        type Query {
          hello: String
        }
      `);
      ...
      

      The type Query defines what queries can be executed and in which format it will return the result. As you can see, the only query defined is hello that returns data in a String format.

      The next section establishes the resolvers, where data is matched to the schemas that you can query:

      index.js

      ...
      // Provide resolver functions for your schema fields
      const resolvers = {
        hello: () => 'Hello world!'
      };
      ...
      

      These resolvers are directly linked to schemas, and return the data that matches these schemas.

      The final part of this code block initializes the GraphQL server, creates the API endpoint with Express, and describes the port on which the GraphQL endpoint is running:

      index.js

      ...
      const app = express();
      app.use('/graphql', graphqlHTTP({
        schema,
        rootValue: resolvers
      }));
      app.listen(4000);
      
      console.log(`🚀 Server ready at http://localhost:4000/graphql`);
      

      After you have added these lines, save and exit from index.js.

      Next, to actually run the GraphQL server you need to run the file index.js with Node.js. This can be done manually from the command line, but it's common practice to set up the package.json file to do this for you.

      Open the package.json file:

      Add the following highlighted line to this file:

      package.json

      {
        "name": "project_name",
        "version": "1.0.0",
        "description": "",
        "main": "index.js",
        "scripts": {
          "start": "node index.js",
          "test": "echo "Error: no test specified" && exit 1"
        },
        "keywords": [],
        "author": "",
        "license": "ISC"
      }
      

      Save and exit the file.

      To start the GraphQL server, execute the following command in the terminal:

      Once you run this, the terminal prompt will disappear, and a message will appear to confirm the GraphQL server is running:

      Output

      🚀 Server ready at http://localhost:4000/graphql

      If you now open up another terminal session, you can test if the GraphQL server is running by executing the following command. This sends a curl POST request with a JSON body after the --data flag that contains your GraphQL query to the local endpoint:

      • curl -X POST -H "Content-Type: application/json" --data '{ "query": "{ hello }" }' http://localhost:4000/graphql

      This will execute the query as it's described in the GraphQL schema in your code and return data in a predictable JSON format that is equal to the data as it's returned in the resolvers:

      Output

      { "data": { "hello": "Hello world!" } }

      Note: In case the Express server crashes or gets stuck, you need to manually kill the node process that is running on the server. To kill all such processes, you can execute the following:

      After which, you can restart the GraphQL server by running:

      In this step you've created the first version of the GraphQL server that is now running on a local endpoint that can be accessed on your server. Next, you'll connect your resolvers to the MongoDB database.

      Step 3 — Connecting to the MongoDB Database

      With the GraphQL server in order, you can now set up the connection with the MongoDB database that you configured and filled with data before and create a new schema that matches this data.

      To be able to connect to MongoDB from the GraphQL server, install the JavaScript package for MongoDB from npm:

      • sudo npm install --save mongodb

      Once this has been installed, open up index.js in your text editor:

      Next, add the following highlighted code to index.js just after the imported dependencies and fill the highlighted values with your own connection details to the local MongoDB database. The username, password, and database_name are those that you created in the first step of this tutorial.

      index.js

      const express = require('express');
      const graphqlHTTP = require('express-graphql');
      const { buildSchema } = require('graphql');
      const { MongoClient } = require('mongodb');
      
      const context = () => MongoClient.connect('mongodb://username:password@localhost:27017/database_name', { useNewUrlParser: true }).then(client => client.db('database_name'));
      ...
      

      These lines add the connection to the local MongoDB database to a function called context. This context function will be available to every resolver, which is why you use this to set up database connections.

      Next, in your index.js file, add the context function to the initialization of the GraphQL server by inserting the following highlighted lines:

      index.js

      ...
      const app = express();
      app.use('/graphql', graphqlHTTP({
        schema,
        rootValue: resolvers,
        context
      }));
      app.listen(4000);
      
      console.log(`🚀 Server ready at http://localhost:4000/graphql`);
      

      Now you can call this context function from your resolvers, and thereby read variables from the MongoDB database. If you look back to the first step of this tutorial, you can see which values are present in the database. From here, define a new GraphQL schema that matches this data structure. Overwrite the previous value for the constant schema with the following highlighted lines:

      index.js

      ...
      // Construct a schema, using GrahQL schema language
      const schema = buildSchema(`
        type Query {
          bios: [Bio]
        }
        type Bio {
          name: Name,
          title: String,
          birth: String,
          death: String,
          awards: [Award]
        }
        type Name {
          first: String,
          last: String
        },
        type Award {
          award: String,
          year: Float,
          by: String
        }
      `);
      ...
      

      The type Query has changed and now returns a collection of the new type Bio. This new type consists of several types including two other non-scalar types Name and Awards, meaning these types don't match a predefined format like String or Float. For more information on defining GraphQL schemas you can look at the documentation for GraphQL.

      Also, since the resolvers tie the data from the database to the schema, update the code for the resolvers when you make changes to the schema. Create a new resolver that is called bios, which is equal to the Query that can be found in the schema and the name of the collection in the database. Note that, in this case, the name of the collection in db.collection('bios') is bios, but that this would change if you had assigned a different name to your collection.

      Add the following highlighted line to index.js:

      index.js

      ...
      // Provide resolver functions for your schema fields
      const resolvers = {
        bios: (args, context) => context().then(db => db.collection('bios').find().toArray())
      };
      ...
      

      This function will use the context function, which you can use to retrieve variables from the MongoDB database. Once you have made these changes to the code, save and exit index.js.

      In order to make these changes active, you need to restart the GraphQL server. You can stop the current process by using the keyboard combination CTRL + C and start the GraphQL server by running:

      Now you're able to use the updated schema and query the data that is inside the database. If you look at the schema, you'll see that the Query for bios returns the type Bio; this type could also return the type Name.

      To return all the first and last names for all the bios in the database, send the following request to the GraphQL server in a new terminal window:

      • curl -X POST -H "Content-Type: application/json" --data '{ "query": "{ bios { name { first, last } } }" }' http://localhost:4000/graphql

      This again will return a JSON object that matches the structure of the schema:

      Output

      {"data":{"bios":[{"name":{"first":"John","last":"Backus"}},{"name":{"first":"John","last":"McCarthy"}}]}}

      You can easily retrieve more variables from the bios by extending the query with any of the types that are described in the type for Bio.

      Also, you can retrieve a bio by specifying an id. In order to do this you need to add another type to the Query type and extend the resolvers. To do this, open index.js in your text editor:

      Add the following highlighted lines of code:

      index.js

      ...
      // Construct a schema, using GrahQL schema language
      const schema = buildSchema(`
        type Query {
          bios: [Bio]
          bio(id: Int): Bio
        }
      
        ...
      
        // Provide resolver functions for your schema fields
        const resolvers = {
          bios: (args, context) => context().then(db => db.collection('bios').find().toArray()),
          bio: (args, context) => context().then(db => db.collection('bios').findOne({ _id: args.id }))
        };
        ...
      

      Save and exit the file.

      In the terminal that is running your GraphQL server, press CTRL + C to stop it from running, then execute the following to restart it:

      In another terminal window, execute the following GraphQL request:

      • curl -X POST -H "Content-Type: application/json" --data '{ "query": "{ bio(id: 1) { name { first, last } } }" }' http://localhost:4000/graphql

      This returns the entry for the bio that has an id equal to 1:

      Output

      { "data": { "bio": { "name": { "first": "John", "last": "Backus" } } } }

      Being able to query data from a database is not the only feature of GraphQL; you can also change the data in the database. To do this, open up index.js:

      Next to the type Query you can also use the type Mutation, which allows you to mutate the database. To use this type, add it to the schema and also create input types by inserting these highlighted lines:

      index.js

      ...
      // Construct a schema, using GraphQL schema language
      const schema = buildSchema(`
        type Query {
          bios: [Bio]
          bio(id: Int): Bio
        }
        type Mutation {
          addBio(input: BioInput) : Bio
        }
        input BioInput {
          name: NameInput
          title: String
          birth: String
          death: String
        }
        input NameInput {
          first: String
          last: String
        }
      ...
      

      These input types define which variables can be used as inputs, which you can access in the resolvers and use to insert a new document in the database. Do this by adding the following lines to index.js:

      index.js

      ...
      // Provide resolver functions for your schema fields
      const resolvers = {
        bios: (args, context) => context().then(db => db.collection('bios').find().toArray()),
        bio: (args, context) => context().then(db => db.collection('bios').findOne({ _id: args.id })),
        addBio: (args, context) => context().then(db => db.collection('bios').insertOne({ name: args.input.name, title: args.input.title, death: args.input.death, birth: args.input.birth})).then(response => response.ops[0])
      };
      ...
      

      Just as with the resolvers for regular queries, you need to return a value from the resolver in index.js. In the case of a Mutation where the type Bio is mutated, you would return the value of the mutated bio.

      At this point, your index.js file will contain the following lines:

      index.js

      iconst express = require('express');
      const graphqlHTTP = require('express-graphql');
      const { buildSchema } = require('graphql');
      const { MongoClient } = require('mongodb');
      
      const context = () => MongoClient.connect('mongodb://username:password@localhost:27017/database_name', { useNewUrlParser: true })
        .then(client => client.db('GraphQL_Test'));
      
      // Construct a schema, using GraphQL schema language
      const schema = buildSchema(`
        type Query {
          bios: [Bio]
          bio(id: Int): Bio
        }
        type Mutation {
          addBio(input: BioInput) : Bio
        }
        input BioInput {
          name: NameInput
          title: String
          birth: String
          death: String
        }
        input NameInput {
          first: String
          last: String
        }
        type Bio {
          name: Name,
          title: String,
          birth: String,
          death: String,
          awards: [Award]
        }
        type Name {
          first: String,
          last: String
        },
        type Award {
          award: String,
          year: Float,
          by: String
        }
      `);
      
      // Provide resolver functions for your schema fields
      const resolvers = {
        bios: (args, context) =>context().then(db => db.collection('Sample_Data').find().toArray()),
        bio: (args, context) =>context().then(db => db.collection('Sample_Data').findOne({ _id: args.id })),
        addBio: (args, context) => context().then(db => db.collection('Sample_Data').insertOne({ name: args.input.name, title: args.input.title, death: args.input.death, birth: args.input.birth})).then(response => response.ops[0])
      };
      
      const app = express();
      app.use('/graphql', graphqlHTTP({
        schema,
        rootValue: resolvers,
        context
      }));
      app.listen(4000);
      
      console.log(`🚀 Server ready at http://localhost:4000/graphql`);
      

      Save and exit index.js.

      To check if your new mutation is working, restart the GraphQL server by pressing CTRL + c and running npm start in the terminal that is running your GraphQL server, then open another terminal session to execute the following curl request. Just as with the curl request for queries, the body in the --data flag will be sent to the GraphQL server. The highlighted parts will be added to the database:

      • curl -X POST -H "Content-Type: application/json" --data '{ "query": "mutation { addBio(input: { name: { first: "test", last: "user" } }) { name { first, last } } }" }' http://localhost:4000/graphql

      This returns the following result, meaning you just inserted a new bio to the database:

      Output

      { "data": { "addBio": { "name": { "first": "test", "last": "user" } } } }

      In this step, you created the connection with MongoDB and the GraphQL server, allowing you to retrieve and mutate data from this database by executing GraphQL queries. Next, you'll expose this GraphQL server for remote access.

      Step 4 — Allowing Remote Access

      Having set up the database and the GraphQL server, you can now configure the GraphQL server to allow remote access. For this you'll use Nginx, which you set up in the prerequisite tutorial How to install Nginx on Ubuntu 18.04. This Nginx configuration can be found in the /etc/nginx/sites-available/example.com file, where example.com is the server name you added in the prerequisite tutorial.

      Open this file for editing, replacing your domain name with example.com:

      • sudo nano /etc/nginx/sites-available/example.com

      In this file you can find a server block that listens to port 80, where you've already set up a value for server_name in the prerequisite tutorial. Inside this server block, change the value for root to be the directory in which you created the code for the GraphQL server and add index.js as the index. Also, within the location block, set a proxy_pass so you can use your server's IP or a custom domain name to refer to the GraphQL server:

      /etc/nginx/sites-available/example.com

      server {
        listen 80;
        listen [::]:80;
      
        root /project_name;
        index index.js;
      
        server_name example.com;
      
        location / {
          proxy_pass http://localhost:4000/graphql;
        }
      }
      

      Make sure there are no Nginx syntax errors in this configuration file by running:

      You will receive the following output:

      Output

      nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful

      When there are no errors found for the configuration file, restart Nginx:

      • sudo systemctl restart nginx

      Now you will be able to access your GraphQL server from any terminal session tab by executing and replacing example.com by either your server's IP or your custom domain name:

      • curl -X POST -H "Content-Type: application/json" --data '{ "query": "{ bios { name { first, last } } }" }' http://example.com

      This will return the same JSON object as the one of the previous step, including any additional data you might have added by using a mutation:

      Output

      {"data":{"bios":[{"name":{"first":"John","last":"Backus"}},{"name":{"first":"John","last":"McCarthy"}},{"name":{"first":"test","last":"user"}}]}}

      Now that you have made your GraphQL server accessible remotely, make sure your GraphQL server doesn't go down when you close the terminal or the server restarts. This way, your MongoDB database will be accessible via the GraphQL server whenever you want to make a request.

      To do this, use the npm package forever, a CLI tool that ensures that your command line scripts run continuously, or get restarted in case of any failure.

      Install forever with npm:

      • sudo npm install forever -g

      Once it is done installing, add it to the package.json file:

      package.json

      {
        "name": "project_name",
        "version": "1.0.0",
        "description": "",
        "main": "index.js",
        "scripts": {
          "start": "node index.js",
          "deploy": "forever start --minUptime 2000 --spinSleepTime 5 index.js",
          "test": "echo "Error: no test specified" && exit 1"
        },
        ...
      

      To start the GraphQL server with forever enabled, run the following command:

      This will start the index.js file containing the GraphQL server with forever, and ensure it will keep running with a minimum uptime of 2000 milliseconds and 5 milliseconds between every restart in case of a failure. The GraphQL server will now continuously run in the background, so you don't need to open a new tab any longer when you want to send a request to the server.

      You've now created a GraphQL server that is using MongoDB to store data and is set up to allow access from a remote server. In the next step you'll enable the GraphiQL playground, which will make it easier for you to inspect the GraphQL server.

      Step 5 — Enabling GraphiQL Playground

      Being able to send cURL requests to the GraphQL server is great, but it would be faster to have a user interface that can execute GraphQL requests immediately, especially during development. For this you can use GraphiQL, an interface supported by the package express-graphql.

      To enable GraphiQL, edit the file index.js:

      Add the following highlighted lines:

      index.js

      const app = express();
      app.use('/graphql', graphqlHTTP({
        schema,
        rootValue: resolvers,
        context,
        graphiql: true
      }));
      app.listen(4000);
      
      console.log(`🚀 Server ready at http://localhost:4000/graphql`);
      

      Save and exit the file.

      In order for these changes to become visible, make sure to stop forever by executing:

      Next, start forever again so the latest version of your GraphQL server is running:

      Open a browser at the URL http://example.com, replacing example.com with your domain name or your server IP. You will see the GraphiQL playground, where you can type GraphQL requests.

      The initial screen for the GraphiQL playground

      On the left side of this playground you can type the GraphQL queries and mutations, while the output will be shown on the right side of the playground. To test if this is working, type the following query on the left side:

      query {
        bios {
          name {
            first
            last
          }
        }
      }
      

      This will output the same result on the right side of the playground, again in JSON format:

      The GraphiQL playground in action

      Now you can send GraphQL requests using the terminal and the GraphiQL playground.

      Conclusion

      In this tutorial you've set up a MongoDB database and retrieved and mutated data from this database using GraphQL, Node.js, and Express for the server. Additionally, you configured Nginx to allow remote access to this server. Not only can you send requests to this GraphQL server directly, you can also use the GraphiQL as a visual, in-browser GraphQL interface.

      If you want to learn about GraphQL, you can watch a recording of my presentation on GraphQL at NDC {London} or visit the website howtographql.com for tutorials about GraphQL. To study how GraphQL interacts with other technologies, check out the tutorial on How to Manually Set Up a Prisma Server on Ubuntu 18.04, and for more information on building applications with MongoDB, see How To Build a Blog with Nest.js, MongoDB, and Vue.js.



      Source link

      How To Apply Computer Vision to Build an Emotion-Based Dog Filter in Python 3


      The author selected Girls Who Code to receive a donation as part of the Write for DOnations program.

      Introduction

      Computer vision is a subfield of computer science that aims to extract a higher-order understanding from images and videos. This field includes tasks such as object detection, image restoration (matrix completion), and optical flow. Computer vision powers technologies such as self-driving car prototypes, employee-less grocery stores, fun Snapchat filters, and your mobile device’s face authenticator.

      In this tutorial, you will explore computer vision as you use pre-trained models to build a Snapchat-esque dog filter. For those unfamiliar with Snapchat, this filter will detect your face and then superimpose a dog mask on it. You will then train a face-emotion classifier so that the filter can pick dog masks based on emotion, such as a corgi for happy or a pug for sad. Along the way, you will also explore related concepts in both ordinary least squares and computer vision, which will expose you to the fundamentals of machine learning.

      A working dog filter

      As you work through the tutorial, you’ll use OpenCV, a computer-vision library, numpy for linear algebra utilities, and matplotlib for plotting. You’ll also apply the following concepts as you build a computer-vision application:

      • Ordinary least squares as a regression and classification technique.
      • The basics of stochastic gradient neural networks.

      While not necessary to complete this tutorial, you’ll find it easier to understand some of the more detailed explanations if you’re familiar with these mathematical concepts:

      • Fundamental linear algebra concepts: scalars, vectors, and matrices.
      • Fundamental calculus: how to take a derivative.

      You can find the complete code for this tutorial at https://github.com/do-community/emotion-based-dog-filter.

      Let’s get started.

      Prerequisites

      To complete this tutorial, you will need the following:

      Step 1 — Creating The Project and Installing Dependencies

      Let’s create a workspace for this project and install the dependencies we’ll need. We’ll call our workspace DogFilter:

      Navigate to the DogFilter directory:

      Then create a new Python virtual environment for the project:

      • python3 -m venv dogfilter

      Activate your environment.

      • source dogfilter/bin/activate

      The prompt changes, indicating the environment is active. Now install PyTorch, a deep-learning framework for Python that we'll use in this tutorial. The installation process depends on which operating system you're using.

      On macOS, install Pytorch with the following command:

      • python -m pip install torch==0.4.1 torchvision==0.2.1

      On Linux, use the following commands:

      • pip install http://download.pytorch.org/whl/cpu/torch-0.4.1-cp35-cp35m-linux_x86_64.whl
      • pip install torchvision

      And for Windows, install Pytorch with these commands:

      • pip install http://download.pytorch.org/whl/cpu/torch-0.4.1-cp35-cp35m-win_amd64.whl
      • pip install torchvision

      Now install prepackaged binaries for OpenCV and numpy, which are computer vision and linear algebra libraries, respectively. The former offers utilities such as image rotations, and the latter offers linear algebra utilities such as a matrix inversion.

      • python -m pip install opencv-python==3.4.3.18 numpy==1.14.5

      Finally, create a directory for our assets, which will hold the images we'll use in this tutorial:

      With the dependencies installed, let's build the first version of our filter: a face detector.

      Step 2 — Building a Face Detector

      Our first objective is to detect all faces in an image. We'll create a script that accepts a single image and outputs an annotated image with the faces outlined with boxes.

      Fortunately, instead of writing our own face detection logic, we can use pre-trained models. We'll set up a model and then load pre-trained parameters. OpenCV makes this easy by providing both.

      OpenCV provides the model parameters in its source code. but we need the absolute path to our locally-installed OpenCV to use these parameters. Since that absolute path may vary, we'll download our own copy instead and place it in the assets folder:

      • wget -O assets/haarcascade_frontalface_default.xml https://github.com/opencv/opencv/raw/master/data/haarcascades/haarcascade_frontalface_default.xml

      The -O option specifies the destination as assets/haarcascade_frontalface_default.xml. The second argument is the source URL.

      We'll detect all faces in the following image from Pexels (CC0, link to original image).

      Picture of children

      First, download the image. The following command saves the downloaded image as children.png in the assets folder:

      • wget -O assets/children.png https://www.xpresservers.com/wp-content/uploads/2019/04/How-To-Apply-Computer-Vision-to-Build-an-Emotion-Based-Dog-Filter-in-Python-3.png

      To check that the detection algorithm works, we will run it on an individual image and save the resulting annotated image to disk. Create an outputs folder for these annotated results.

      Now create a Python script for the face detector. Create the file step_1_face_detect using nano or your favorite text editor:

      • nano step_2_face_detect.py

      Add the following code to the file. This code imports OpenCV, which contains the image utilities and face classifier. The rest of the code is typical Python program boilerplate.

      step_2_face_detect.py

      """Test for face detection"""
      
      import cv2
      
      
      def main():
          pass
      
      if __name__ == '__main__':
          main()
      

      Now replace pass in the main function with this code which initializes a face classifier using the OpenCV parameters you downloaded to your assets folder:

      step_2_face_detect.py

      def main():
          # initialize front face classifier
          cascade = cv2.CascadeClassifier("assets/haarcascade_frontalface_default.xml")
      

      Next, add this line to load the image children.png.

      step_2_face_detect.py

          frame = cv2.imread('assets/children.png')
      

      Then add this code to convert the image to black and white, as the classifier was trained on black-and-white images. To accomplish this, we convert to grayscale and then discretize the histogram:

      step_2_face_detect.py

          # Convert to black-and-white
          gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
          blackwhite = cv2.equalizeHist(gray)
      

      Then use OpenCV's detectMultiScale function to detect all faces in the image.

      step_2_face_detect.py

          rects = cascade.detectMultiScale(
              blackwhite, scaleFactor=1.3, minNeighbors=4, minSize=(30, 30),
              flags=cv2.CASCADE_SCALE_IMAGE)
      
      • scaleFactor specifies how much the image is reduced along each dimension.
      • minNeighbors denotes how many neighboring rectangles a candidate rectangle needs to be retained.
      • minSize is the minimum allowable detected object size. Objects smaller than this are discarded.

      The return type is a list of tuples, where each tuple has four numbers denoting the minimum x, minimum y, width, and height of the rectangle in that order.

      Iterate over all detected objects and draw them on the image in green using cv2.rectangle:

      step_2_face_detect.py

          for x, y, w, h in rects:
              cv2.rectangle(frame, (x, y), (x + w, y + h), (0, 255, 0), 2)
      
      • The second and third arguments are opposing corners of the rectangle.
      • The fourth argument is the color to use. (0, 255, 0) corresponds to green for our RGB color space.
      • The last argument denotes the width of our line.

      Finally, write the image with bounding boxes into a new file at outputs/children_detected.png:

      step_2_face_detect.py

          cv2.imwrite('outputs/children_detected.png', frame)
      

      Your completed script should look like this:

      step_2_face_detect.py

      """Tests face detection for a static image."""  
      
      import cv2  
      
      
      def main():  
      
          # initialize front face classifier  
          cascade = cv2.CascadeClassifier(  
              "assets/haarcascade_frontalface_default.xml")  
      
          frame = cv2.imread('assets/children.png')  
      
          # Convert to black-and-white  
          gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)  
          blackwhite = cv2.equalizeHist(gray)  
      
          rects = cascade.detectMultiScale(  
              blackwhite, scaleFactor=1.3, minNeighbors=4, minSize=(30, 30),  
          flags=cv2.CASCADE_SCALE_IMAGE)  
      
          for x, y, w, h in rects:  
              cv2.rectangle(frame, (x, y), (x + w, y + h), (0, 255, 0), 2)  
      
          cv2.imwrite('outputs/children_detected.png', frame)  
      
      if __name__ == '__main__':  
          main()
      

      Save the file and exit your editor. Then run the script:

      • python step_2_face_detect.py

      Open outputs/children_detected.png. You'll see the following image that shows the faces outlined with boxes:

      Picture of children with bounding boxes

      At this point, you have a working face detector. It accepts an image as input and draws bounding boxes around all faces in the image, outputting the annotated image. Now let's apply this same detection to a live camera feed.

      Step 3 — Linking the Camera Feed

      The next objective is to link the computer's camera to the face detector. Instead of detecting faces in a static image, you'll detect all faces from your computer's camera. You will collect camera input, detect and annotate all faces, and then display the annotated image back to the user. You'll continue from the script in Step 2, so start by duplicating that script:

      • cp step_2_face_detect.py step_3_camera_face_detect.py

      Then open the new script in your editor:

      • nano step_3_camera_face_detect.py

      You will update the main function by using some elements from this test script from the official OpenCV documentation. Start by initializing a VideoCapture object that is set to capture live feed from your computer's camera. Place this at the start of the main function, before the other code in the function:

      step_3_camera_face_detect.py

      def main():
          cap = cv2.VideoCapture(0)
          ...
      

      Starting from the line defining frame, indent all of your existing code, placing all of the code in a while loop.

      step_3_camera_face_detect.py

          while True:
              frame = cv2.imread('assets/children.png')
              ...
              for x, y, w, h in rects:  
                  cv2.rectangle(frame, (x, y), (x + w, y + h), (0, 255, 0), 2)  
      
              cv2.imwrite('outputs/children_detected.png', frame)
      

      Replace the line defining frame at the start of the while loop. Instead of reading from an image on disk, you're now reading from the camera:

      step_3_camera_face_detect.py

          while True:
              # frame = cv2.imread('assets/children.png') # DELETE ME
              # Capture frame-by-frame
              ret, frame = cap.read()
      

      Replace the line cv2.imwrite(...) at the end of the while loop. Instead of writing an image to disk, you'll display the annotated image back to the user's screen:

      step_3_camera_face_detect.py

            cv2.imwrite('outputs/children_detected.png', frame)  # DELETE ME
            # Display the resulting frame
            cv2.imshow('frame', frame)
      

      Also, add some code to watch for keyboard input so you can stop the program. Check if the user hits the q character and, if so, quit the application. Right after cv2.imshow(...) add the following:

      step_3_camera_face_detect.py

      ...
              cv2.imshow('frame', frame)
              if cv2.waitKey(1) & 0xFF == ord('q'):
                  break
      ...
      

      The line cv2.waitkey(1) halts the program for 1 millisecond so that the captured image can be displayed back to the user.

      Finally, release the capture and close all windows. Place this outside of the while loop to end the main function.

      step_3_camera_face_detect.py

      ...
      
          while True:
          ...
      
      
          cap.release()
          cv2.destroyAllWindows()
      

      Your script should look like the following:

      step_3_camera_face_detect.py

      """Test for face detection on video camera.
      
      Move your face around and a green box will identify your face.
      With the test frame in focus, hit `q` to exit.
      Note that typing `q` into your terminal will do nothing.
      """
      
      import cv2
      
      
      def main():
          cap = cv2.VideoCapture(0)
      
          # initialize front face classifier
          cascade = cv2.CascadeClassifier(
              "assets/haarcascade_frontalface_default.xml")
      
          while True:
              # Capture frame-by-frame
              ret, frame = cap.read()
      
              # Convert to black-and-white
              gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
              blackwhite = cv2.equalizeHist(gray)
      
              # Detect faces
              rects = cascade.detectMultiScale(
                  blackwhite, scaleFactor=1.3, minNeighbors=4, minSize=(30, 30),
                  flags=cv2.CASCADE_SCALE_IMAGE)
      
              # Add all bounding boxes to the image
              for x, y, w, h in rects:
                  cv2.rectangle(frame, (x, y), (x + w, y + h), (0, 255, 0), 2)
      
              # Display the resulting frame
              cv2.imshow('frame', frame)
              if cv2.waitKey(1) & 0xFF == ord('q'):
                  break
      
          # When everything done, release the capture
          cap.release()
          cv2.destroyAllWindows()
      
      
      if __name__ == '__main__':
          main()
      

      Save the file and exit your editor.

      Now run the test script.

      • python step_3_camera_face_detect.py

      This activates your camera and opens a window displaying your camera's feed. Your face will be boxed by a green square in real time:

      Working face detector

      Note: If you find that you have to hold very still for things to work, the lighting in the room may not be adequate. Try moving to a brightly lit room where you and your background have high constrast. Also, avoid bright lights near your head. For example, if you have your back to the sun, this process might not work very well.

      Our next objective is to take the detected faces and superimpose dog masks on each one.

      Step 4 — Building the Dog Filter

      Before we build the filter itself, let's explore how images are represented numerically. This will give you the background needed to modify images and ultimately apply a dog filter.

      Let's look at an example. We can construct a black-and-white image using numbers, where 0 corresponds to black and 1 corresponds to white.

      Focus on the dividing line between 1s and 0s. What shape do you see?

      0 0 0 0 0 0 0 0 0
      0 0 0 0 1 0 0 0 0
      0 0 0 1 1 1 0 0 0
      0 0 1 1 1 1 1 0 0
      0 0 0 1 1 1 0 0 0
      0 0 0 0 1 0 0 0 0
      0 0 0 0 0 0 0 0 0
      

      The image is a diamond. If save this matrix of values as an image. This gives us the following picture:

      Diamond as picture

      We can use any value between 0 and 1, such as 0.1, 0.26, or 0.74391. Numbers closer to 0 are darker and numbers closer to 1 are lighter. This allows us to represent white, black, and any shade of gray. This is great news for us because we can now construct any grayscale image using 0, 1, and any value in between. Consider the following, for example. Can you tell what it is? Again, each number corresponds to the color of a pixel.

      1  1  1  1  1  1  1  1  1  1  1  1
      1  1  1  1  0  0  0  0  1  1  1  1
      1  1  0  0 .4 .4 .4 .4  0  0  1  1
      1  0 .4 .4 .5 .4 .4 .4 .4 .4  0  1
      1  0 .4 .5 .5 .5 .4 .4 .4 .4  0  1
      0 .4 .4 .4 .5 .4 .4 .4 .4 .4 .4  0
      0 .4 .4 .4 .4  0  0 .4 .4 .4 .4  0
      0  0 .4 .4  0  1 .7  0 .4 .4  0  0
      0  1  0  0  0 .7 .7  0  0  0  1  0
      1  0  1  1  1  0  0 .7 .7 .4  0  1
      1  0 .7  1  1  1 .7 .7 .7 .7  0  1
      1  1  0  0 .7 .7 .7 .7  0  0  1  1
      1  1  1  1  0  0  0  0  1  1  1  1
      1  1  1  1  1  1  1  1  1  1  1  1
      

      Re-rendered as an image, you can now tell that this is, in fact, a Poké Ball:

      Pokeball as picture

      You've now seen how black-and-white and grayscale images are represented numerically. To introduce color, we need a way to encode more information. An image has its height and width expressed as h x w.

      Image

      In the current grayscale representation, each pixel is one value between 0 and 1. We can equivalently say our image has dimensions h x w x 1. In other words, every (x, y) position in our image has just one value.

      Grayscale image

      For a color representation, we represent the color of each pixel using three values between 0 and 1. One number corresponds to the "degree of red," one to the "degree of green," and the last to the "degree of blue." We call this the RGB color space. This means that for every (x, y) position in our image, we have three values (r, g, b). As a result, our image is now h x w x 3:

      Color image

      Here, each number ranges from 0 to 255 instead of 0 to 1, but the idea is the same. Different combinations of numbers correspond to different colors, such as dark purple (102, 0, 204) or bright orange (255, 153, 51). The takeaways are as follows:

      1. Each image will be represented as a box of numbers that has three dimensions: height, width, and color channels. Manipulating this box of numbers directly is equivalent to manipulating the image.
      2. We can also flatten this box to become just a list of numbers. In this way, our image becomes a vector. Later on, we will refer to images as vectors.

      Now that you understand how images are represented numerically, you are well-equipped to begin applying dog masks to faces. To apply a dog mask, you will replace values in the child image with non-white dog mask pixels. To start, you will work with a single image. Download this crop of a face from the image you used in Step 2.

      • wget -O assets/child.png https://www.xpresservers.com/wp-content/uploads/2019/04/1554419826_451_How-To-Apply-Computer-Vision-to-Build-an-Emotion-Based-Dog-Filter-in-Python-3.png

      Cropped face

      Additionally, download the following dog mask. The dog masks used in this tutorial are my own drawings, now released to the public domain under a CC0 License.

      Dog mask

      Download this with wget:

      • wget -O assets/dog.png https://www.xpresservers.com/wp-content/uploads/2019/04/1554419826_685_How-To-Apply-Computer-Vision-to-Build-an-Emotion-Based-Dog-Filter-in-Python-3.png

      Create a new file called step_4_dog_mask_simple.py which will hold the code for the script that applies the dog mask to faces:

      • nano step_4_dog_mask_simple.py

      Add the following boilerplate for the Python script and import the OpenCV and numpy libraries:

      step_4_dog_mask_simple.py

      """Test for adding dog mask"""
      
      import cv2
      import numpy as np
      
      
      def main():
          pass
      
      if __name__ == '__main__':
          main()
      

      Replace pass in the main function with these two lines which load the original image and the dog mask into memory.

      step_4_dog_mask_simple.py

      ...
      def main():
          face = cv2.imread('assets/child.png')
          mask = cv2.imread('assets/dog.png')
      

      Next, fit the dog mask to the child. The logic is more complicated than what we've done previously, so we will create a new function called apply_mask to modularize our code. Directly after the two lines that load the images, add this line which invokes the apply_mask function:

      step_4_dog_mask_simple.py

      ...
          face_with_mask = apply_mask(face, mask)
      

      Create a new function called apply_mask and place it above the main function:

      step_4_dog_mask_simple.py

      ...
      def apply_mask(face: np.array, mask: np.array) -> np.array:
          """Add the mask to the provided face, and return the face with mask."""
          pass
      
      def main():
      ...
      

      At this point, your file should look like this:

      step_4_dog_mask_simple.py

      """Test for adding dog mask"""
      
      import cv2
      import numpy as np
      
      
      def apply_mask(face: np.array, mask: np.array) -> np.array:
          """Add the mask to the provided face, and return the face with mask."""
          pass
      
      
      def main():
          face = cv2.imread('assets/child.png')
          mask = cv2.imread('assets/dog.png')
          face_with_mask = apply_mask(face, mask)
      
      if __name__ == '__main__':
          main()
      

      Let's build out the apply_mask function. Our goal is to apply the mask to the child's face. However, we need to maintain the aspect ratio for our dog mask. To do so, we need to explicitly compute our dog mask's final dimensions. Inside the apply_mask function, replace pass with these two lines which extract the height and width of both images:

      step_4_dog_mask_simple.py

      ...
          mask_h, mask_w, _ = mask.shape
          face_h, face_w, _ = face.shape
      

      Next, determine which dimension needs to be "shrunk more." To be precise, we need the tighter of the two constraints. Add this line to the apply_mask function:

      step_4_dog_mask_simple.py

      ...
      
          # Resize the mask to fit on face
          factor = min(face_h / mask_h, face_w / mask_w)
      

      Then compute the new shape by adding this code to the function:

      step_4_dog_mask_simple.py

      ...
          new_mask_w = int(factor * mask_w)
          new_mask_h = int(factor * mask_h)
          new_mask_shape = (new_mask_w, new_mask_h)
      

      Here we cast the numbers to integers, as the resize function needs integral dimensions.

      Now add this code to resize the dog mask to the new shape:

      step_4_dog_mask_simple.py

      ...
      
          # Add mask to face - ensure mask is centered
          resized_mask = cv2.resize(mask, new_mask_shape)
      

      Finally, write the image to disk so you can double-check that your resized dog mask is correct after you run the script:

      step_4_dog_mask_simple.py

          cv2.imwrite('outputs/resized_dog.png', resized_mask)
      

      The completed script should look like this:

      step_4_dog_mask_simple.py

      """Test for adding dog mask"""
      import cv2
      import numpy as np
      
      def apply_mask(face: np.array, mask: np.array) -> np.array:
          """Add the mask to the provided face, and return the face with mask."""
          mask_h, mask_w, _ = mask.shape
          face_h, face_w, _ = face.shape
      
          # Resize the mask to fit on face
          factor = min(face_h / mask_h, face_w / mask_w)
          new_mask_w = int(factor * mask_w)
          new_mask_h = int(factor * mask_h)
          new_mask_shape = (new_mask_w, new_mask_h)
      
          # Add mask to face - ensure mask is centered
          resized_mask = cv2.resize(mask, new_mask_shape)
          cv2.imwrite('outputs/resized_dog.png', resized_mask)
      
      
      def main():
          face = cv2.imread('assets/child.png')
          mask = cv2.imread('assets/dog.png')
          face_with_mask = apply_mask(face, mask)
      
      if __name__ == '__main__':
          main()
      
      

      Save the file and exit your editor. Run the new script:

      • python step_4_dog_mask_simple.py

      Open the image at outputs/resized_dog.png to double-check the mask was resized correctly. It will match the dog mask shown earlier in this section.

      Now add the dog mask to the child. Open the step_4_dog_mask_simple.py file again and return to the apply_mask function:

      • nano step_4_dog_mask_simple.py

      First, remove the line of code that writes the resized mask from the apply_mask function since you no longer need it:

          cv2.imwrite('outputs/resized_dog.png', resized_mask)  # delete this line
          ...
      

      In its place, apply your knowledge of image representation from the start of this section to modify the image. Start by making a copy of the child image. Add this line to the apply_mask function:

      step_4_dog_mask_simple.py

      ...
          face_with_mask = face.copy()
      

      Next, find all positions where the dog mask is not white or near white. To do this, check if the pixel value is less than 250 across all color channels, as we'd expect a near-white pixel to be near [255, 255, 255]. Add this code:

      step_4_dog_mask_simple.py

      ...
          non_white_pixels = (resized_mask < 250).all(axis=2)
      

      At this point, the dog image is, at most, as large as the child image. We want to center the dog image on the face, so compute the offset needed to center the dog image by adding this code to apply_mask:

      step_4_dog_mask_simple.py

      ...
          off_h = int((face_h - new_mask_h) / 2)  
          off_w = int((face_w - new_mask_w) / 2)
      

      Copy all non-white pixels from the dog image into the child image. Since the child image may be larger than the dog image, we need to take a subset of the child image:

      step_4_dog_mask_simple.py

          face_with_mask[off_h: off_h+new_mask_h, off_w: off_w+new_mask_w][non_white_pixels] = 
                  resized_mask[non_white_pixels]
      

      Then return the result:

      step_4_dog_mask_simple.py

          return face_with_mask
      

      In the main function, add this code to write the result of the apply_mask function to an output image so you can manually double-check the result:

      step_4_dog_mask_simple.py

      ...
          face_with_mask = apply_mask(face, mask)
          cv2.imwrite('outputs/child_with_dog_mask.png', face_with_mask)
      

      Your completed script will look like the following:

      step_4_dog_mask_simple.py

      """Test for adding dog mask"""
      
      import cv2
      import numpy as np
      
      
      def apply_mask(face: np.array, mask: np.array) -> np.array:
          """Add the mask to the provided face, and return the face with mask."""
          mask_h, mask_w, _ = mask.shape
          face_h, face_w, _ = face.shape
      
          # Resize the mask to fit on face
          factor = min(face_h / mask_h, face_w / mask_w)
          new_mask_w = int(factor * mask_w)
          new_mask_h = int(factor * mask_h)
          new_mask_shape = (new_mask_w, new_mask_h)
          resized_mask = cv2.resize(mask, new_mask_shape)
      
          # Add mask to face - ensure mask is centered
          face_with_mask = face.copy()
          non_white_pixels = (resized_mask < 250).all(axis=2)
          off_h = int((face_h - new_mask_h) / 2)  
          off_w = int((face_w - new_mask_w) / 2)
          face_with_mask[off_h: off_h+new_mask_h, off_w: off_w+new_mask_w][non_white_pixels] = 
               resized_mask[non_white_pixels]
      
          return face_with_mask
      
      def main():
          face = cv2.imread('assets/child.png')
          mask = cv2.imread('assets/dog.png')
          face_with_mask = apply_mask(face, mask)
          cv2.imwrite('outputs/child_with_dog_mask.png', face_with_mask)
      
      if __name__ == '__main__':
          main()
      

      Save the script and run it:

      • python step_4_dog_mask_simple.py

      You'll have the following picture of a child with a dog mask in outputs/child_with_dog_mask.png:

      Picture of child with dog mask on

      You now have a utility that applies dog masks to faces. Now let's use what you've built to add the dog mask in real time.

      We'll pick up from where we left off in Step 3. Copy step_3_camera_face_detect.py to step_4_dog_mask.py.

      • cp step_3_camera_face_detect.py step_4_dog_mask.py

      Open your new script.

      First, import the NumPy library at the top of the script:

      step_4_dog_mask.py

      import numpy as np
      ...
      

      Then add the apply_mask function from your previous work into this new file above the main function:

      step_4_dog_mask.py

      def apply_mask(face: np.array, mask: np.array) -> np.array:
          """Add the mask to the provided face, and return the face with mask."""
          mask_h, mask_w, _ = mask.shape
          face_h, face_w, _ = face.shape
      
          # Resize the mask to fit on face
          factor = min(face_h / mask_h, face_w / mask_w)
          new_mask_w = int(factor * mask_w)
          new_mask_h = int(factor * mask_h)
          new_mask_shape = (new_mask_w, new_mask_h)
          resized_mask = cv2.resize(mask, new_mask_shape)
      
          # Add mask to face - ensure mask is centered
          face_with_mask = face.copy()
          non_white_pixels = (resized_mask < 250).all(axis=2)
          off_h = int((face_h - new_mask_h) / 2)  
          off_w = int((face_w - new_mask_w) / 2)
          face_with_mask[off_h: off_h+new_mask_h, off_w: off_w+new_mask_w][non_white_pixels] = 
               resized_mask[non_white_pixels]
      
          return face_with_mask
      ...
      

      Second, locate this line in the main function:

      step_4_dog_mask.py

          cap = cv2.VideoCapture(0)
      

      Add this code after that line to load the dog mask:

      step_4_dog_mask.py

          cap = cv2.VideoCapture(0)
      
          # load mask
          mask = cv2.imread('assets/dog.png')
          ...
      

      Next, in the while loop, locate this line:

      step_4_dog_mask.py

              ret, frame = cap.read()
      

      Add this line after it to extract the image's height and width:

      step_4_dog_mask.py

              ret, frame = cap.read()
              frame_h, frame_w, _ = frame.shape
              ...
      

      Next, delete the line in main that draws bounding boxes. You'll find this line in the for loop that iterates over detected faces:

      step_4_dog_mask.py

              for x, y, w, h in rects:
              ...
                  cv2.rectangle(frame, (x, y), (x + w, y + h), (0, 255, 0), 2) # DELETE ME
              ...
      

      In its place, add this code which crops the frame. For aesthetic purposes, we crop an area slightly larger than the face.

      step_4_dog_mask.py

              for x, y, w, h in rects:
                  # crop a frame slightly larger than the face
                  y0, y1 = int(y - 0.25*h), int(y + 0.75*h)
                  x0, x1 = x, x + w
      

      Introduce a check in case the detected face is too close to the edge.

      step_4_dog_mask.py

                  # give up if the cropped frame would be out-of-bounds
                  if x0 < 0 or y0 < 0 or x1 > frame_w or y1 > frame_h:
                      continue
      

      Finally, insert the face with a mask into the image.

      step_4_dog_mask.py

                  # apply mask
                  frame[y0: y1, x0: x1] = apply_mask(frame[y0: y1, x0: x1], mask)
      

      Verify that your script looks like this:

      step_4_dog_mask.py

      """Real-time dog filter
      
      Move your face around and a dog filter will be applied to your face if it is not out-of-bounds. With the test frame in focus, hit `q` to exit. Note that typing `q` into your terminal will do nothing.
      """
      
      import numpy as np
      import cv2
      
      
      def apply_mask(face: np.array, mask: np.array) -> np.array:
          """Add the mask to the provided face, and return the face with mask."""
          mask_h, mask_w, _ = mask.shape
          face_h, face_w, _ = face.shape
      
          # Resize the mask to fit on face
          factor = min(face_h / mask_h, face_w / mask_w)
          new_mask_w = int(factor * mask_w)
          new_mask_h = int(factor * mask_h)
          new_mask_shape = (new_mask_w, new_mask_h)
          resized_mask = cv2.resize(mask, new_mask_shape)
      
          # Add mask to face - ensure mask is centered
          face_with_mask = face.copy()
          non_white_pixels = (resized_mask < 250).all(axis=2)
          off_h = int((face_h - new_mask_h) / 2)
          off_w = int((face_w - new_mask_w) / 2)
          face_with_mask[off_h: off_h+new_mask_h, off_w: off_w+new_mask_w][non_white_pixels] = 
               resized_mask[non_white_pixels]
      
          return face_with_mask
      
      def main():
          cap = cv2.VideoCapture(0)
      
          # load mask
          mask = cv2.imread('assets/dog.png')
      
          # initialize front face classifier
          cascade = cv2.CascadeClassifier("assets/haarcascade_frontalface_default.xml")
      
          while(True):
              # Capture frame-by-frame
              ret, frame = cap.read()
              frame_h, frame_w, _ = frame.shape
      
              # Convert to black-and-white
              gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
              blackwhite = cv2.equalizeHist(gray)
      
              # Detect faces
              rects = cascade.detectMultiScale(
                  blackwhite, scaleFactor=1.3, minNeighbors=4, minSize=(30, 30),
                  flags=cv2.CASCADE_SCALE_IMAGE)
      
              # Add mask to faces
              for x, y, w, h in rects:
                  # crop a frame slightly larger than the face
                  y0, y1 = int(y - 0.25*h), int(y + 0.75*h)
                  x0, x1 = x, x + w
      
                  # give up if the cropped frame would be out-of-bounds
                  if x0 < 0 or y0 < 0 or x1 > frame_w or y1 > frame_h:
                      continue
      
                  # apply mask
                  frame[y0: y1, x0: x1] = apply_mask(frame[y0: y1, x0: x1], mask)
      
              # Display the resulting frame
              cv2.imshow('frame', frame)
              if cv2.waitKey(1) & 0xFF == ord('q'):
                  break
      
          # When everything done, release the capture
          cap.release()
          cv2.destroyAllWindows()
      
      
      if __name__ == '__main__':
          main()
      

      Save the file and exit your editor. Then run the script.

      • python step_4_dog_mask.py

      You now have a real-time dog filter running. The script will also work with multiple faces in the picture, so you can get your friends together for some automatic dog-ification.

      GIF for working dog filter

      This concludes our first primary objective in this tutorial, which is to create a Snapchat-esque dog filter. Now let's use facial expression to determine the dog mask applied to a face.

      Step 5 — Build a Basic Face Emotion Classifier using Least Squares

      In this section you'll create an emotion classifier to apply different masks based on displayed emotions. If you smile, the filter will apply a corgi mask. If you frown, it will apply a pug mask. Along the way, you'll explore the least-squares framework, which is fundamental to understanding and discussing machine learning concepts.

      To understand how to process our data and produce predictions, we'll first briefly explore machine learning models.

      We need to ask two questions for each model that we consider. For now, these two questions will be sufficient to differentiate between models:

      1. Input: What information is the model given?
      2. Output: What is the model trying to predict?

      At a high-level, the goal is to develop a model for emotion classification. The model is:

      1. Input: given images of faces.
      2. Output: predicts the corresponding emotion.
      model: face -> emotion
      

      The approach we'll use is least squares; we take a set of points, and we find a line of best fit. The line of best fit, shown in the following image, is our model.

      Least Squares

      Consider the input and output for our line:

      1. Input: given x coordinates.
      2. Output: predicts the corresponding $y$ coordinate.
      least squares line: x -> y
      

      Our input x must represent faces and our output y must represent emotion, in order for us to use least squares for emotion classification:

      • x -> face: Instead of using one number for x, we will use a vector of values for x. Thus, x can represent images of faces. The article Ordinary Least Squares explains why you can use a vector of values for x.
      • y -> emotion: Each emotion will correspond to a number. For example, "angry" is 0, "sad" is 1, and "happy" is 2. In this way, y can represent emotions. However, our line is not constrained to output the y values 0, 1, and 2. It has an infinite number of possible y values–it could be 1.2, 3.5, or 10003.42. How do we translate those y values to integers corresponding to classes? See the article One-Hot Encoding for more detail and explanation.

      Armed with this background knowledge, you will build a simple least-squares classifier using vectorized images and one-hot encoded labels. You'll accomplish this in three steps:

      1. Preprocess the data: As explained at the start of this section, our samples are vectors where each vector encodes an image of a face. Our labels are integers corresponding to an emotion, and we'll apply one-hot encoding to these labels.
      2. Specify and train the model: Use the closed-form least squares solution, w^*.
      3. Run a prediction using the model: Take the argmax of Xw^* to obtain predicted emotions.

      Let's get started.

      First, set up a directory to contain the data:

      Then download the data, curated by Pierre-Luc Carrier and Aaron Courville, from a 2013 Face Emotion Classification competition on Kaggle.

      • wget -O data/fer2013.tar https://bitbucket.org/alvinwan/adversarial-examples-in-computer-vision-building-then-fooling/raw/babfe4651f89a398c4b3fdbdd6d7a697c5104cff/fer2013.tar

      Navigate to the data directory and unpack the data.

      • cd data
      • tar -xzf fer2013.tar

      Now we'll create a script to run the least-squares model. Navigate to the root of your project:

      Create a new file for the script:

      Add Python boilerplate and import the packages you will need:

      step_5_ls_simple.py

      """Train emotion classifier using least squares."""
      
      import numpy as np
      
      def main():
          pass
      
      if __name__ == '__main__':
          main()
      

      Next, load the data into memory. Replace pass in your main function with the following code:

      step_5_ls_simple.py

      
          # load data
          with np.load('data/fer2013_train.npz') as data:
              X_train, Y_train = data['X'], data['Y']
      
          with np.load('data/fer2013_test.npz') as data:
              X_test, Y_test = data['X'], data['Y']
      

      Now one-hot encode the labels. To do this, construct the identity matrix with numpy and then index into this matrix using our list of labels:

      step_5_ls_simple.py

          # one-hot labels
          I = np.eye(6)
          Y_oh_train, Y_oh_test = I[Y_train], I[Y_test]
      

      Here, we use the fact that the i-th row in the identity matrix is all zero, except for the i-th entry. Thus, the i-th row is the one-hot encoding for the label of class i. Additionally, we use numpy's advanced indexing, where [a, b, c, d][[1, 3]] = [b, d].

      Computing (X^TX)^{-1} would take too long on commodity hardware, as X^TX is a 2304x2304 matrix with over four million values, so we'll reduce this time by selecting only the first 100 features. Add this code:

      step_5_ls_simple.py

      ...
          # select first 100 dimensions
          A_train, A_test = X_train[:, :100], X_test[:, :100]
      

      Next, add this code to evaluate the closed-form least-squares solution:

      step_5_ls_simple.py

      ...
          # train model
          w = np.linalg.inv(A_train.T.dot(A_train)).dot(A_train.T.dot(Y_oh_train))
      

      Then define an evaluation function for training and validation sets. Place this before your main function:

      step_5_ls_simple.py

      def evaluate(A, Y, w):
          Yhat = np.argmax(A.dot(w), axis=1)
          return np.sum(Yhat == Y) / Y.shape[0]
      

      To estimate labels, we take the inner product with each sample and get the indices of the maximum values using np.argmax. Then we compute the average number of correct classifications. This final number is your accuracy.

      Finally, add this code to the end of the main function to compute the training and validation accuracy using the evaluate function you just wrote:

      step_5_ls_simple.py

          # evaluate model
          ols_train_accuracy = evaluate(A_train, Y_train, w)
          print('(ols) Train Accuracy:', ols_train_accuracy)
          ols_test_accuracy = evaluate(A_test, Y_test, w)
          print('(ols) Test Accuracy:', ols_test_accuracy)
      

      Double-check that your script matches the following:

      step_5_ls_simple.py

      """Train emotion classifier using least squares."""
      
      import numpy as np
      
      
      def evaluate(A, Y, w):
          Yhat = np.argmax(A.dot(w), axis=1)
          return np.sum(Yhat == Y) / Y.shape[0]
      
      def main():
      
          # load data
          with np.load('data/fer2013_train.npz') as data:
              X_train, Y_train = data['X'], data['Y']
      
          with np.load('data/fer2013_test.npz') as data:
              X_test, Y_test = data['X'], data['Y']
      
          # one-hot labels
          I = np.eye(6)
          Y_oh_train, Y_oh_test = I[Y_train], I[Y_test]
      
          # select first 100 dimensions
          A_train, A_test = X_train[:, :100], X_test[:, :100]
      
          # train model
          w = np.linalg.inv(A_train.T.dot(A_train)).dot(A_train.T.dot(Y_oh_train))
      
          # evaluate model
          ols_train_accuracy = evaluate(A_train, Y_train, w)
          print('(ols) Train Accuracy:', ols_train_accuracy)
          ols_test_accuracy = evaluate(A_test, Y_test, w)
          print('(ols) Test Accuracy:', ols_test_accuracy)
      
      
      if __name__ == '__main__':
          main()
      

      Save your file, exit your editor, and run the Python script.

      • python step_5_ls_simple.py

      You'll see the following output:

      Output

      (ols) Train Accuracy: 0.4748918316507146 (ols) Test Accuracy: 0.45280545359202934

      Our model gives 47.5% train accuracy. We repeat this on the validation set to obtain 45.3% accuracy. For a three-way classification problem, 45.3% is reasonably above guessing, which is 33%​. This is our starting classifier for emotion detection, and in the next step, you'll build off of this least-squares model to improve accuracy. The higher the accuracy, the more reliably your emotion-based dog filter can find the appropriate dog filter for each detected emotion.

      Step 6 — Improving Accuracy by Featurizing the Inputs

      We can use a more expressive model to boost accuracy. To accomplish this, we featurize our inputs.

      The original image tells us that position (0, 0) is red, (1, 0) is brown, and so on. A featurized image may tell us that there is a dog to the top-left of the image, a person in the middle, etc. Featurization is powerful, but its precise definition is beyond the scope of this tutorial.

      We'll use an approximation for the radial basis function (RBF) kernel, using a random Gaussian matrix. We won't go into detail in this tutorial. Instead, we'll treat this as a black box that computes higher-order features for us.

      We'll continue where we left off in the previous step. Copy the previous script so you have a good starting point:

      • cp step_5_ls_simple.py step_6_ls_simple.py

      Open the new file in your editor:

      We'll start by creating the featurizing random matrix. Again, we'll use only 100 features in our new feature space.

      Locate the following line, defining A_train and A_test:

      step_6_ls_simple.py

          # select first 100 dimensions
          A_train, A_test = X_train[:, :100], X_test[:, :100]
      

      Directly above this definition for A_train and A_test, add a random feature matrix:

      step_6_ls_simple.py

          d = 100
          W = np.random.normal(size=(X_train.shape[1], d))
          # select first 100 dimensions
          A_train, A_test = X_train[:, :100], X_test[:, :100]  ...
      

      Then replace the definitions for A_train and A_test. We redefine our matrices, called design matrices, using this random featurization.

      step_6_ls_simple.py

          A_train, A_test = X_train.dot(W), X_test.dot(W)
      

      Save your file and run the script.

      • python step_6_ls_simple.py

      You'll see the following output:

      Output

      (ols) Train Accuracy: 0.584174642717 (ols) Test Accuracy: 0.584425799685

      This featurization now offers 58.4% train accuracy and 58.4% validation accuracy, a 13.1% improvement in validation results. We trimmed the X matrix to be 100 x 100, but the choice of 100 was arbirtary. We could also trim the X matrix to be 1000 x 1000 or 50 x 50. Say the dimension of x is d x d. We can test more values of d by re-trimming X to be d x d and recomputing a new model.

      Trying more values of d, we find an additional 4.3% improvement in test accuracy to 61.7%. In the following figure, we consider the performance of our new classifier as we vary d. Intuitively, as d increases, the accuracy should also increase, as we use more and more of our original data. Rather than paint a rosy picture, however, the graph exhibits a negative trend:

      Performance of featurized ordinary least squares

      As we keep more of our data, the gap between the training and validation accuracies increases as well. This is clear evidence of overfitting, where our model is learning representations that are no longer generalizable to all data. To combat overfitting, we'll regularize our model by penalizing complex models.

      We amend our ordinary least-squares objective function with a regularization term, giving us a new objective. Our new objective function is called ridge regression and it looks like this:

      min_w |Aw- y|^2 + lambda |w|^2
      

      In this equation, lambda is a tunable hyperparameter. Plug lambda = 0 into the equation and ridge regression becomes least-squares. Plug lambda = infinity into the equation, and you'll find the best w must now be zero, as any non-zero w incurs infinite loss. As it turns out, this objective yields a closed-form solution as well:

      w^* = (A^TA + lambda I)^{-1}A^Ty
      

      Still using the featurized samples, retrain and reevaluate the model once more.

      Open step_6_ls_simple.py again in your editor:

      This time, increase the dimensionality of the new feature space to d=1000​. Change the value of d from 100 to 1000 as shown in the following code block:

      step_6_ls_simple.py

      ...
          d = 1000
          W = np.random.normal(size=(X_train.shape[1], d))
      ...
      

      Then apply ridge regression using a regularization of lambda = 10^{10}. Replace the line defining w with the following two lines:

      step_6_ls_simple.py

      ...
          # train model
          I = np.eye(A_train.shape[1])
          w = np.linalg.inv(A_train.T.dot(A_train) + 1e10 * I).dot(A_train.T.dot(Y_oh_train))
      

      Then locate this block:

      step_6_ls_simple.py

      ...
        ols_train_accuracy = evaluate(A_train, Y_train, w)
        print('(ols) Train Accuracy:', ols_train_accuracy)
        ols_test_accuracy = evaluate(A_test, Y_test, w)
        print('(ols) Test Accuracy:', ols_test_accuracy)
      

      Replace it with the following:

      step_6_ls_simple.py

      ...
      
        print('(ridge) Train Accuracy:', evaluate(A_train, Y_train, w))
        print('(ridge) Test Accuracy:', evaluate(A_test, Y_test, w))
      

      The completed script should look like this:

      step_6_ls_simple.py

      """Train emotion classifier using least squares."""
      
      import numpy as np
      
      def evaluate(A, Y, w):
          Yhat = np.argmax(A.dot(w), axis=1)
          return np.sum(Yhat == Y) / Y.shape[0]
      
      def main():
          # load data
          with np.load('data/fer2013_train.npz') as data:
              X_train, Y_train = data['X'], data['Y']
      
          with np.load('data/fer2013_test.npz') as data:
              X_test, Y_test = data['X'], data['Y']
      
          # one-hot labels
          I = np.eye(6)
          Y_oh_train, Y_oh_test = I[Y_train], I[Y_test]
          d = 1000
          W = np.random.normal(size=(X_train.shape[1], d))
          # select first 100 dimensions
          A_train, A_test = X_train.dot(W), X_test.dot(W)
      
          # train model
          I = np.eye(A_train.shape[1])
          w = np.linalg.inv(A_train.T.dot(A_train) + 1e10 * I).dot(A_train.T.dot(Y_oh_train))
      
          # evaluate model
          print('(ridge) Train Accuracy:', evaluate(A_train, Y_train, w))
          print('(ridge) Test Accuracy:', evaluate(A_test, Y_test, w))
      
      if __name__ == '__main__':
          main()
      

      Save the file, exit your editor, and run the script:

      • python step_6_ls_simple.py

      You'll see the following output:

      Output

      (ridge) Train Accuracy: 0.651173462698 (ridge) Test Accuracy: 0.622181436812

      There's an additional improvement of 0.4% in validation accuracy to 62.2%, as train accuracy drops to 65.1%. Once again reevaluating across a number of different d, we see a smaller gap between training and validation accuracies for ridge regression. In other words, ridge regression was subject to less overfitting.

      Performance of featurized ols and ridge regression

      Baseline performance for least squares, with these extra enhancements, performs reasonably well. The training and inference times, all together, take no more than 20 seconds for even the best results. In the next section, you'll explore even more complex models.

      Step 7 — Building the Face-Emotion Classifier Using a Convolutional Neural Network in PyTorch

      In this section, you'll build a second emotion classifier using neural networks instead of least squares. Again, our goal is to produce a model that accepts faces as input and outputs an emotion. Eventually, this classifier will then determine which dog mask to apply.

      For a brief neural network visualization and introduction, see the article Understanding Neural Networks. Here, we will use a deep-learning library called PyTorch. There are a number of deep-learning libraries in widespread use, and each has various pros and cons. PyTorch is a particularly good place to start. To impliment this neural network classifier, we again take three steps, as we did with the least-squares classifier:

      1. Preprocess the data: Apply one-hot encoding and then apply PyTorch abstractions.
      2. Specify and train the model: Set up a neural network using PyTorch layers. Define optimization hyperparameters and run stochastic gradient descent.
      3. Run a prediction using the model: Evaluate the neural network.

      Create a new file, named step_7_fer_simple.py

      • nano step_7_fer_simple.py

      Import the necessary utilities and create a Python class that will hold your data. For data processing here, you will create the train and test datasets. To do these, implement PyTorch's Dataset interface, which lets you load and use PyTorch's built-in data pipeline for the face-emotion recognition dataset:

      step_7_fer_simple.py

      from torch.utils.data import Dataset
      from torch.autograd import Variable
      import torch.nn as nn
      import torch.nn.functional as F
      import torch.optim as optim
      import numpy as np
      import torch
      import cv2
      import argparse
      
      
      class Fer2013Dataset(Dataset):
          """Face Emotion Recognition dataset.
      
          Utility for loading FER into PyTorch. Dataset curated by Pierre-Luc Carrier
          and Aaron Courville in 2013.
      
          Each sample is 1 x 1 x 48 x 48, and each label is a scalar.
          """
          pass
      

      Delete the pass placeholder in the Fer2013Dataset class. In its place, add a function that will initialize our data holder:

      step_7_fer_simple.py

          def __init__(self, path: str):
              """
              Args:
                  path: Path to `.np` file containing sample nxd and label nx1
              """
              with np.load(path) as data:
                  self._samples = data['X']
                  self._labels = data['Y']
              self._samples = self._samples.reshape((-1, 1, 48, 48))
      
              self.X = Variable(torch.from_numpy(self._samples)).float()
              self.Y = Variable(torch.from_numpy(self._labels)).float()
      ...
      

      This function starts by loading the samples and labels. Then it wraps the data in PyTorch data structures.

      Directly after the __init__ function, add a __len__ function, as this is needed to implement the Dataset interface PyTorch expects:

      step_7_fer_simple.py

      ...
          def __len__(self):
              return len(self._labels)
      

      Finally, add a __getitem__ method, which returns a dictionary containing the sample and the label:

      step_7_fer_simple.py

          def __getitem__(self, idx):
              return {'image': self._samples[idx], 'label': self._labels[idx]}
      

      Double-check that your file looks like the following:

      step_7_fer_simple.py

      from torch.utils.data import Dataset
      from torch.autograd import Variable
      import torch.nn as nn
      import torch.nn.functional as F
      import torch.optim as optim
      import numpy as np
      import torch
      import cv2
      import argparse
      
      
      class Fer2013Dataset(Dataset):
          """Face Emotion Recognition dataset.
          Utility for loading FER into PyTorch. Dataset curated by Pierre-Luc Carrier
          and Aaron Courville in 2013.
          Each sample is 1 x 1 x 48 x 48, and each label is a scalar.
          """
      
          def __init__(self, path: str):
              """
              Args:
                  path: Path to `.np` file containing sample nxd and label nx1
              """
              with np.load(path) as data:
                  self._samples = data['X']
                  self._labels = data['Y']
              self._samples = self._samples.reshape((-1, 1, 48, 48))
      
              self.X = Variable(torch.from_numpy(self._samples)).float()
              self.Y = Variable(torch.from_numpy(self._labels)).float()
      
          def __len__(self):
              return len(self._labels)
      
          def __getitem__(self, idx):
              return {'image': self._samples[idx], 'label': self._labels[idx]}
      

      Next, load the Fer2013Dataset dataset. Add the following code to the end of your file after the Fer2013Dataset class:

      step_7_fer_simple.py

      trainset = Fer2013Dataset('data/fer2013_train.npz')
      trainloader = torch.utils.data.DataLoader(trainset, batch_size=32, shuffle=True)
      
      testset = Fer2013Dataset('data/fer2013_test.npz')
      testloader = torch.utils.data.DataLoader(testset, batch_size=32, shuffle=False)
      

      This code initializes the dataset using the Fer2013Dataset class you created. Then for the train and validation sets, it wraps the dataset in a DataLoader. This translates the dataset into an iterable to use later.

      As a sanity check, verify that the dataset utilities are functioning. Create a sample dataset loader using DataLoader and print the first element of that loader. Add the following to the end of your file:

      step_7_fer_simple.py

      if __name__ == '__main__':
          loader = torch.utils.data.DataLoader(trainset, batch_size=2, shuffle=False)
          print(next(iter(loader)))
      

      Verify that your completed script looks like this:

      step_7_fer_simple.py

      from torch.utils.data import Dataset
      from torch.autograd import Variable
      import torch.nn as nn
      import torch.nn.functional as F
      import torch.optim as optim
      import numpy as np
      import torch
      import cv2
      import argparse
      
      
      class Fer2013Dataset(Dataset):
          """Face Emotion Recognition dataset.
          Utility for loading FER into PyTorch. Dataset curated by Pierre-Luc Carrier
          and Aaron Courville in 2013.
          Each sample is 1 x 1 x 48 x 48, and each label is a scalar.
          """
      
          def __init__(self, path: str):
              """
              Args:
                  path: Path to `.np` file containing sample nxd and label nx1
              """
              with np.load(path) as data:
                  self._samples = data['X']
                  self._labels = data['Y']
              self._samples = self._samples.reshape((-1, 1, 48, 48))
      
              self.X = Variable(torch.from_numpy(self._samples)).float()
              self.Y = Variable(torch.from_numpy(self._labels)).float()
      
          def __len__(self):
              return len(self._labels)
      
          def __getitem__(self, idx):
              return {'image': self._samples[idx], 'label': self._labels[idx]}
      
      trainset = Fer2013Dataset('data/fer2013_train.npz')
      trainloader = torch.utils.data.DataLoader(trainset, batch_size=32, shuffle=True)
      
      testset = Fer2013Dataset('data/fer2013_test.npz')
      testloader = torch.utils.data.DataLoader(testset, batch_size=32, shuffle=False)
      
      if __name__ == '__main__':
          loader = torch.utils.data.DataLoader(trainset, batch_size=2, shuffle=False)
          print(next(iter(loader)))
      

      Exit your editor and run the script.

      • python step_7_fer_simple.py

      This outputs the following pair of tensors. Our data pipeline outputs two samples and two labels. This indicates that our data pipeline is up and ready to go:

      Output

      {'image': (0 ,0 ,.,.) = 24 32 36 ... 173 172 173 25 34 29 ... 173 172 173 26 29 25 ... 172 172 174 ... ⋱ ... 159 185 157 ... 157 156 153 136 157 187 ... 152 152 150 145 130 161 ... 142 143 142 ⋮ (1 ,0 ,.,.) = 20 17 19 ... 187 176 162 22 17 17 ... 195 180 171 17 17 18 ... 203 193 175 ... ⋱ ... 1 1 1 ... 106 115 119 2 2 1 ... 103 111 119 2 2 2 ... 99 107 118 [torch.LongTensor of size 2x1x48x48] , 'label': 1 1 [torch.LongTensor of size 2] }

      Now that you've verified that the data pipeline works, return to step_7_fer_simple.py to add the neural network and optimizer. Open step_7_fer_simple.py.

      • nano step_7_fer_simple.py

      First, delete the last three lines you added in the previous iteration:

      step_7_fer_simple.py

      # Delete all three lines
      if __name__ == '__main__':
          loader = torch.utils.data.DataLoader(trainset, batch_size=2, shuffle=False)
          print(next(iter(loader)))
      

      In their place, define a PyTorch neural network that includes three convolutional layers, followed by three fully connected layers. Add this to the end of your existing script:

      step_7_fer_simple.py

      class Net(nn.Module):
          def __init__(self):
              super(Net, self).__init__()
              self.conv1 = nn.Conv2d(1, 6, 5)
              self.pool = nn.MaxPool2d(2, 2)
              self.conv2 = nn.Conv2d(6, 6, 3)
              self.conv3 = nn.Conv2d(6, 16, 3)
              self.fc1 = nn.Linear(16 * 4 * 4, 120)
              self.fc2 = nn.Linear(120, 48)
              self.fc3 = nn.Linear(48, 3)
      
          def forward(self, x):
              x = self.pool(F.relu(self.conv1(x)))
              x = self.pool(F.relu(self.conv2(x)))
              x = self.pool(F.relu(self.conv3(x)))
              x = x.view(-1, 16 * 4 * 4)
              x = F.relu(self.fc1(x))
              x = F.relu(self.fc2(x))
              x = self.fc3(x)
              return x
      

      Now initialize the neural network, define a loss function, and define optimization hyperparameters by adding the following code to the end of the script:

      step_7_fer_simple.py

      net = Net().float()
      criterion = nn.CrossEntropyLoss()
      optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
      

      We'll train for two epochs. For now, we define an epoch to be an iteration of training where every training sample has been used exactly once.

      First, extract image and label from the dataset loader and then wrap each in a PyTorch Variable. Second, run the forward pass and then backpropagate through the loss and neural network. Add the following code to the end of your script to do that:

      step_7_fer_simple.py

      for epoch in range(2):  # loop over the dataset multiple times
      
          running_loss = 0.0
          for i, data in enumerate(trainloader, 0):
              inputs = Variable(data['image'].float())
              labels = Variable(data['label'].long())
              optimizer.zero_grad()
      
              # forward + backward + optimize
              outputs = net(inputs)
              loss = criterion(outputs, labels)
              loss.backward()
              optimizer.step()
      
              # print statistics
              running_loss += loss.data[0]
              if i % 100 == 0:
                  print('[%d, %5d] loss: %.3f' % (epoch, i, running_loss / (i + 1)))
      

      Your script should now look like this:

      step_7_fer_simple.py

      from torch.utils.data import Dataset
      from torch.autograd import Variable
      import torch.nn as nn
      import torch.nn.functional as F
      import torch.optim as optim
      import numpy as np
      import torch
      import cv2
      import argparse
      
      
      class Fer2013Dataset(Dataset):
          """Face Emotion Recognition dataset.
      
          Utility for loading FER into PyTorch. Dataset curated by Pierre-Luc Carrier
          and Aaron Courville in 2013.
      
          Each sample is 1 x 1 x 48 x 48, and each label is a scalar.
          """
          def __init__(self, path: str):
              """
              Args:
                  path: Path to `.np` file containing sample nxd and label nx1
              """
              with np.load(path) as data:
                  self._samples = data['X']
                  self._labels = data['Y']
              self._samples = self._samples.reshape((-1, 1, 48, 48))
      
              self.X = Variable(torch.from_numpy(self._samples)).float()
              self.Y = Variable(torch.from_numpy(self._labels)).float()
      
          def __len__(self):
              return len(self._labels)
      
      
          def __getitem__(self, idx):
              return {'image': self._samples[idx], 'label': self._labels[idx]}
      
      
      trainset = Fer2013Dataset('data/fer2013_train.npz')
      trainloader = torch.utils.data.DataLoader(trainset, batch_size=32, shuffle=True)
      
      testset = Fer2013Dataset('data/fer2013_test.npz')
      testloader = torch.utils.data.DataLoader(testset, batch_size=32, shuffle=False)
      
      
      class Net(nn.Module):
          def __init__(self):
              super(Net, self).__init__()
              self.conv1 = nn.Conv2d(1, 6, 5)
              self.pool = nn.MaxPool2d(2, 2)
              self.conv2 = nn.Conv2d(6, 6, 3)
              self.conv3 = nn.Conv2d(6, 16, 3)
              self.fc1 = nn.Linear(16 * 4 * 4, 120)
              self.fc2 = nn.Linear(120, 48)
              self.fc3 = nn.Linear(48, 3)
      
          def forward(self, x):
              x = self.pool(F.relu(self.conv1(x)))
              x = self.pool(F.relu(self.conv2(x)))
              x = self.pool(F.relu(self.conv3(x)))
              x = x.view(-1, 16 * 4 * 4)
              x = F.relu(self.fc1(x))
              x = F.relu(self.fc2(x))
              x = self.fc3(x)
              return x
      
      net = Net().float()
      criterion = nn.CrossEntropyLoss()
      optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
      
      
      for epoch in range(2):  # loop over the dataset multiple times
      
          running_loss = 0.0
          for i, data in enumerate(trainloader, 0):
              inputs = Variable(data['image'].float())
              labels = Variable(data['label'].long())
              optimizer.zero_grad()
      
              # forward + backward + optimize
              outputs = net(inputs)
              loss = criterion(outputs, labels)
              loss.backward()
              optimizer.step()
      
              # print statistics
              running_loss += loss.data[0]
              if i % 100 == 0:
                  print('[%d, %5d] loss: %.3f' % (epoch, i, running_loss / (i + 1)))
      

      Save the file and exit the editor once you've verified your code. Then, launch this proof-of-concept training:

      • python step_7_fer_simple.py

      You'll see output similar to the following as the neural network trains:

      Output

      [0, 0] loss: 1.094 [0, 100] loss: 1.049 [0, 200] loss: 1.009 [0, 300] loss: 0.963 [0, 400] loss: 0.935 [1, 0] loss: 0.760 [1, 100] loss: 0.768 [1, 200] loss: 0.775 [1, 300] loss: 0.776 [1, 400] loss: 0.767

      You can then augment this script using a number of other PyTorch utilities to save and load models, output training and validation accuracies, fine-tune a learning-rate schedule, etc. After training for 20 epochs with a learning rate of 0.01 and momentum of 0.9, our neural network attains a 87.9% train accuracy and a 75.5% validation accuracy, a further 6.8% improvement over the most successful least-squares approach thus far at 66.6%. We'll include these additional bells and whistles in a new script.

      Create a new file to hold the final face emotion detector which your live camera feed will use. This script contains the code above along with a command-line interface and an easy-to-import version of our code that will be used later. Additionally, it contains the hyperparameters tuned in advance, for a model with higher accuracy.

      Start with the following imports. This matches our previous file but additionally includes OpenCV as import cv2.

      step_7_fer.py

      from torch.utils.data import Dataset
      from torch.autograd import Variable
      import torch.nn as nn
      import torch.nn.functional as F
      import torch.optim as optim
      import numpy as np
      import torch
      import cv2
      import argparse
      

      Directly beneath these imports, reuse your code from step_7_fer_simple.py to define the neural network:

      step_7_fer.py

      class Net(nn.Module):
          def __init__(self):
              super(Net, self).__init__()
              self.conv1 = nn.Conv2d(1, 6, 5)
              self.pool = nn.MaxPool2d(2, 2)
              self.conv2 = nn.Conv2d(6, 6, 3)
              self.conv3 = nn.Conv2d(6, 16, 3)
              self.fc1 = nn.Linear(16 * 4 * 4, 120)
              self.fc2 = nn.Linear(120, 48)
              self.fc3 = nn.Linear(48, 3)
      
          def forward(self, x):
              x = self.pool(F.relu(self.conv1(x)))
              x = self.pool(F.relu(self.conv2(x)))
              x = self.pool(F.relu(self.conv3(x)))
              x = x.view(-1, 16 * 4 * 4)
              x = F.relu(self.fc1(x))
              x = F.relu(self.fc2(x))
              x = self.fc3(x)
              return x
      

      Again, reuse the code for the Face Emotion Recognition dataset from step_7_fer_simple.py and add it to this file:

      step_7_fer.py

      class Fer2013Dataset(Dataset):
          """Face Emotion Recognition dataset.
          Utility for loading FER into PyTorch. Dataset curated by Pierre-Luc Carrier
          and Aaron Courville in 2013.
          Each sample is 1 x 1 x 48 x 48, and each label is a scalar.
          """
      
          def __init__(self, path: str):
              """
              Args:
                  path: Path to `.np` file containing sample nxd and label nx1
              """
              with np.load(path) as data:
                  self._samples = data['X']
                  self._labels = data['Y']
              self._samples = self._samples.reshape((-1, 1, 48, 48))
      
              self.X = Variable(torch.from_numpy(self._samples)).float()
              self.Y = Variable(torch.from_numpy(self._labels)).float()
      
          def __len__(self):
              return len(self._labels)
      
          def __getitem__(self, idx):
              return {'image': self._samples[idx], 'label': self._labels[idx]}
      

      Next, define a few utilities to evaluate the neural network's performance. First, add an evaluate function which compares the neural network's predicted emotion to the true emotion for a single image:

      step_7_fer.py

      def evaluate(outputs: Variable, labels: Variable, normalized: bool=True) -> float:
          """Evaluate neural network outputs against non-one-hotted labels."""
          Y = labels.data.numpy()
          Yhat = np.argmax(outputs.data.numpy(), axis=1)
          denom = Y.shape[0] if normalized else 1
          return float(np.sum(Yhat == Y) / denom)
      

      Then add a function called batch_evaluate which applies the first function to all images:

      step_7_fer.py

      def batch_evaluate(net: Net, dataset: Dataset, batch_size: int=500) -> float:
          """Evaluate neural network in batches, if dataset is too large."""
          score = 0.0
          n = dataset.X.shape[0]
          for i in range(0, n, batch_size):
              x = dataset.X[i: i + batch_size]
              y = dataset.Y[i: i + batch_size]
              score += evaluate(net(x), y, False)
          return score / n
      

      Now, define a function called get_image_to_emotion_predictor that takes in an image and outputs a predicted emotion, using a pretrained model:

      step_7_fer.py

      def get_image_to_emotion_predictor(model_path='assets/model_best.pth'):
          """Returns predictor, from image to emotion index."""
          net = Net().float()
          pretrained_model = torch.load(model_path)
          net.load_state_dict(pretrained_model['state_dict'])
      
          def predictor(image: np.array):
              """Translates images into emotion indices."""
              if image.shape[2] > 1:
                  image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
              frame = cv2.resize(image, (48, 48)).reshape((1, 1, 48, 48))
              X = Variable(torch.from_numpy(frame)).float()
              return np.argmax(net(X).data.numpy(), axis=1)[0]
          return predictor
      

      Finally, add the following code to define the main function to leverage the other utilities:

      step_7_fer.py

      def main():
          trainset = Fer2013Dataset('data/fer2013_train.npz')
          testset = Fer2013Dataset('data/fer2013_test.npz')
          net = Net().float()
      
          pretrained_model = torch.load("assets/model_best.pth")
          net.load_state_dict(pretrained_model['state_dict'])
      
          train_acc = batch_evaluate(net, trainset, batch_size=500)
          print('Training accuracy: %.3f' % train_acc)
          test_acc = batch_evaluate(net, testset, batch_size=500)
          print('Validation accuracy: %.3f' % test_acc)
      
      
      if __name__ == '__main__':
          main()
      

      This loads a pretrained neural network and evaluates its performance on the provided Face Emotion Recognition dataset. Specifically, the script outputs accuracy on the images we used for training, as well as a separate set of images we put aside for testing purposes.

      Double-check that your file matches the following:

      step_7_fer.py

      from torch.utils.data import Dataset
      from torch.autograd import Variable
      import torch.nn as nn
      import torch.nn.functional as F
      import torch.optim as optim
      import numpy as np
      import torch
      import cv2
      import argparse
      
      class Net(nn.Module):
          def __init__(self):
              super(Net, self).__init__()
              self.conv1 = nn.Conv2d(1, 6, 5)
              self.pool = nn.MaxPool2d(2, 2)
              self.conv2 = nn.Conv2d(6, 6, 3)
              self.conv3 = nn.Conv2d(6, 16, 3)
              self.fc1 = nn.Linear(16 * 4 * 4, 120)
              self.fc2 = nn.Linear(120, 48)
              self.fc3 = nn.Linear(48, 3)
      
          def forward(self, x):
              x = self.pool(F.relu(self.conv1(x)))
              x = self.pool(F.relu(self.conv2(x)))
              x = self.pool(F.relu(self.conv3(x)))
              x = x.view(-1, 16 * 4 * 4)
              x = F.relu(self.fc1(x))
              x = F.relu(self.fc2(x))
              x = self.fc3(x)
              return x
      
      
      class Fer2013Dataset(Dataset):
          """Face Emotion Recognition dataset.
          Utility for loading FER into PyTorch. Dataset curated by Pierre-Luc Carrier
          and Aaron Courville in 2013.
          Each sample is 1 x 1 x 48 x 48, and each label is a scalar.
          """
      
          def __init__(self, path: str):
              """
              Args:
                  path: Path to `.np` file containing sample nxd and label nx1
              """
              with np.load(path) as data:
                  self._samples = data['X']
                  self._labels = data['Y']
              self._samples = self._samples.reshape((-1, 1, 48, 48))
      
              self.X = Variable(torch.from_numpy(self._samples)).float()
              self.Y = Variable(torch.from_numpy(self._labels)).float()
      
          def __len__(self):
              return len(self._labels)
      
          def __getitem__(self, idx):
              return {'image': self._samples[idx], 'label': self._labels[idx]}
      
      
      def evaluate(outputs: Variable, labels: Variable, normalized: bool=True) -> float:
          """Evaluate neural network outputs against non-one-hotted labels."""
          Y = labels.data.numpy()
          Yhat = np.argmax(outputs.data.numpy(), axis=1)
          denom = Y.shape[0] if normalized else 1
          return float(np.sum(Yhat == Y) / denom)
      
      
      def batch_evaluate(net: Net, dataset: Dataset, batch_size: int=500) -> float:
          """Evaluate neural network in batches, if dataset is too large."""
          score = 0.0
          n = dataset.X.shape[0]
          for i in range(0, n, batch_size):
              x = dataset.X[i: i + batch_size]
              y = dataset.Y[i: i + batch_size]
              score += evaluate(net(x), y, False)
          return score / n
      
      
      def get_image_to_emotion_predictor(model_path='assets/model_best.pth'):
          """Returns predictor, from image to emotion index."""
          net = Net().float()
          pretrained_model = torch.load(model_path)
          net.load_state_dict(pretrained_model['state_dict'])
      
          def predictor(image: np.array):
              """Translates images into emotion indices."""
              if image.shape[2] > 1:
                  image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
              frame = cv2.resize(image, (48, 48)).reshape((1, 1, 48, 48))
              X = Variable(torch.from_numpy(frame)).float()
              return np.argmax(net(X).data.numpy(), axis=1)[0]
          return predictor
      
      
      def main():
          trainset = Fer2013Dataset('data/fer2013_train.npz')
          testset = Fer2013Dataset('data/fer2013_test.npz')
          net = Net().float()
      
          pretrained_model = torch.load("assets/model_best.pth")
          net.load_state_dict(pretrained_model['state_dict'])
      
          train_acc = batch_evaluate(net, trainset, batch_size=500)
          print('Training accuracy: %.3f' % train_acc)
          test_acc = batch_evaluate(net, testset, batch_size=500)
          print('Validation accuracy: %.3f' % test_acc)
      
      
      if __name__ == '__main__':
          main(
      

      Save the file and exit your editor.

      As before, with the face detector, download pre-trained model parameters and save them to your assets folder with the following command:

      • wget -O assets/model_best.pth https://github.com/alvinwan/emotion-based-dog-filter/raw/master/src/assets/model_best.pth

      Run the script to use and evaluate the pre-trained model:

      This will output the following:

      Output

      Training accuracy: 0.879 Validation accuracy: 0.755

      At this point, you've built a pretty accurate face-emotion classifier. In essence, our model can correctly disambiguate between faces that are happy, sad, and surprised eight out of ten times. This is a reasonably good model, so you can now move on to using this face-emotion classifier to determine which dog mask to apply to faces.

      Step 8 — Finishing the Emotion-Based Dog Filter

      Before integrating our brand-new face-emotion classifier, we will need animal masks to pick from. We'll use a Dalmation mask and a Sheepdog mask:

      Dalmation mask
      Sheepdog mask

      Execute these commands to download both masks to your assets folder:

      • wget -O assets/dalmation.png https://www.xpresservers.com/wp-content/uploads/2019/04/1554419827_591_How-To-Apply-Computer-Vision-to-Build-an-Emotion-Based-Dog-Filter-in-Python-3.png # dalmation
      • wget -O assets/sheepdog.png https://www.xpresservers.com/wp-content/uploads/2019/04/1554419827_102_How-To-Apply-Computer-Vision-to-Build-an-Emotion-Based-Dog-Filter-in-Python-3.png # sheepdog

      Now let's use the masks in our filter. Start by duplicating the step_4_dog_mask.py file:

      • cp step_4_dog_mask.py step_8_dog_emotion_mask.py

      Open the new Python script.

      • nano step_8_dog_emotion_mask.py

      Insert a new line at the top of the script to import the emotion predictor:

      step_8_dog_emotion_mask.py

      from step_7_fer import get_image_to_emotion_predictor
      ...
      

      Then, in the main() function, locate this line:

      step_8_dog_emotion_mask.py

          mask = cv2.imread('assets/dog.png')
      

      Replace it with the following to load the new masks and aggregate all masks into a tuple:

      step_8_dog_emotion_mask.py

          mask0 = cv2.imread('assets/dog.png')
          mask1 = cv2.imread('assets/dalmation.png')
          mask2 = cv2.imread('assets/sheepdog.png')
          masks = (mask0, mask1, mask2)
      

      Add a line break, and then add this code to create the emotion predictor.

      step_8_dog_emotion_mask.py

      
          # get emotion predictor
          predictor = get_image_to_emotion_predictor()
      

      Your main function should now match the following:

      step_8_dog_emotion_mask.py

      def main():
          cap = cv2.VideoCapture(0)
      
          # load mask
          mask0 = cv2.imread('assets/dog.png')
          mask1 = cv2.imread('assets/dalmation.png')
          mask2 = cv2.imread('assets/sheepdog.png')
          masks = (mask0, mask1, mask2)
      
          # get emotion predictor
          predictor = get_image_to_emotion_predictor()
      
          # initialize front face classifier
          ...
      

      Next, locate these lines:

      step_8_dog_emotion_mask.py

      
                  # apply mask
                  frame[y0: y1, x0: x1] = apply_mask(frame[y0: y1, x0: x1], mask)
      

      Insert the following line below the # apply mask line to select the appropriate mask by using the predictor:

      step_8_dog_emotion_mask.py

                  # apply mask
                  mask = masks[predictor(frame[y:y+h, x: x+w])]
                  frame[y0: y1, x0: x1] = apply_mask(frame[y0: y1, x0: x1], mask)
      
      

      The completed file should look like this:

      step_8_dog_emotion_mask.py

      """Test for face detection"""
      
      from step_7_fer import get_image_to_emotion_predictor
      import numpy as np
      import cv2
      
      def apply_mask(face: np.array, mask: np.array) -> np.array:
          """Add the mask to the provided face, and return the face with mask."""
          mask_h, mask_w, _ = mask.shape
          face_h, face_w, _ = face.shape
      
          # Resize the mask to fit on face
          factor = min(face_h / mask_h, face_w / mask_w)
          new_mask_w = int(factor * mask_w)
          new_mask_h = int(factor * mask_h)
          new_mask_shape = (new_mask_w, new_mask_h)
          resized_mask = cv2.resize(mask, new_mask_shape)
      
          # Add mask to face - ensure mask is centered
          face_with_mask = face.copy()
          non_white_pixels = (resized_mask < 250).all(axis=2)
          off_h = int((face_h - new_mask_h) / 2)
          off_w = int((face_w - new_mask_w) / 2)
          face_with_mask[off_h: off_h+new_mask_h, off_w: off_w+new_mask_w][non_white_pixels] = 
               resized_mask[non_white_pixels]
      
          return face_with_mask
      
      def main():
      
          cap = cv2.VideoCapture(0)
          # load mask
          mask0 = cv2.imread('assets/dog.png')
          mask1 = cv2.imread('assets/dalmation.png')
          mask2 = cv2.imread('assets/sheepdog.png')
          masks = (mask0, mask1, mask2)
      
          # get emotion predictor
          predictor = get_image_to_emotion_predictor()
      
          # initialize front face classifier
          cascade = cv2.CascadeClassifier("assets/haarcascade_frontalface_default.xml")
      
          while True:
              # Capture frame-by-frame
              ret, frame = cap.read()
              frame_h, frame_w, _ = frame.shape
      
              # Convert to black-and-white
              gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
              blackwhite = cv2.equalizeHist(gray)
      
              rects = cascade.detectMultiScale(
                  blackwhite, scaleFactor=1.3, minNeighbors=4, minSize=(30, 30),
                  flags=cv2.CASCADE_SCALE_IMAGE)
      
              for x, y, w, h in rects:
                  # crop a frame slightly larger than the face
                  y0, y1 = int(y - 0.25*h), int(y + 0.75*h)
                  x0, x1 = x, x + w
                  # give up if the cropped frame would be out-of-bounds
                  if x0 < 0 or y0 < 0 or x1 > frame_w or y1 > frame_h:
                      continue
                  # apply mask
                  mask = masks[predictor(frame[y:y+h, x: x+w])]
                  frame[y0: y1, x0: x1] = apply_mask(frame[y0: y1, x0: x1], mask)
      
              # Display the resulting frame
              cv2.imshow('frame', frame)
              if cv2.waitKey(1) & 0xFF == ord('q'):
                  break
      
          cap.release()
          cv2.destroyAllWindows()
      
      if __name__ == '__main__':
          main()
      

      Save and exit your editor. Now launch the script:

      • python step_8_dog_emotion_mask.py

      Now try it out! Smiling will register as "happy" and show the original dog. A neutral face or a frown will register as "sad" and yield the dalmation. A face of "surprise," with a nice big jaw drop, will yield the sheepdog.

      GIF for emotion-based dog filter

      This concludes our emotion-based dog filter and foray into computer vision.

      Conclusion

      In this tutorial, you built a face detector and dog filter using computer vision and employed machine learning models to apply masks based on detected emotions.

      Machine learning is widely applicable. However, it's up to the practitioner to consider the ethical implications of each application when applying machine learning. The application you built in this tutorial was a fun exercise, but remember that you relied on OpenCV and an existing dataset to identify faces, rather than supplying your own data to train the models. The data and models used have significant impacts on how a program works.

      For example, imagine a job search engine where the models were trained with data about candidates. such as race, gender, age, culture, first language, or other factors. And perhaps the developers trained a model that enforces sparsity, which ends up reducing the feature space to a subspace where gender explains most of the variance. As a result, the model influences candidate job searches and even company selection processes based primarily on gender. Now consider more complex situations where the model is less interpretable and you don't know what a particular feature corresponds to. You can learn more about this in Equality of Opportunity in Machine Learning by Professor Moritz Hardt at UC Berkeley.

      There can be an overwhelming magnitude of uncertainty in machine learning. To understand this randomness and complexity, you'll have to develop both mathematical intuitions and probabilistic thinking skills. As a practitioner, it is up to you to dig into the theoretical underpinnings of machine learning.



      Source link

      How To Build and Deploy a Flask Application Using Docker on Ubuntu 18.04


      The author selected the Tech Education Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Docker is an open-source application that allows administrators to create, manage, deploy, and replicate applications using containers. Containers can be thought of as a package that houses dependencies that an application requires to run at an operating system level. This means that each application deployed using Docker lives in an environment of its own and its requirements are handled separately.

      Flask is a web micro-framework that is built on Python. It is called a micro-framework because it does not require specific tools or plug-ins to run. The Flask framework is lightweight and flexible, yet highly structured, making it preferred over other frameworks.

      Deploying a Flask application with Docker will allow you to replicate the application across different servers with minimal reconfiguration.

      In this tutorial, you will create a Flask application and deploy it with Docker. This tutorial will also cover how to update an application after deployment.

      Prerequisites

      To follow this tutorial, you will need the following:

      Step 1 — Setting Up the Flask Application

      To get started, you will create a directory structure that will hold your Flask application. This tutorial will create a directory called TestApp in /var/www, but you can modify the command to name it whatever you’d like.

      • sudo mkdir /var/www/TestApp

      Move in to the newly created TestApp directory:

      Next, create the base folder structure for the Flask application:

      • sudo mkdir -p app/static app/templates

      The -p flag indicates that mkdir will create a directory and all parent directories that don't exist. In this case, mkdir will create the app parent directory in the process of making the static and templates directories.

      The app directory will contain all files related to the Flask application such as its views and blueprints. Views are the code you write to respond to requests to your application. Blueprints create application components and support common patterns within an application or across multiple applications.

      The static directory is where assets such as images, CSS, and JavaScript files live. The templates directory is where you will put the HTML templates for your project.

      Now that the base folder structure is complete, create the files needed to run the Flask application. First, create an __init__.py file inside the app directory. This file tells the Python interpreter that the app directory is a package and should be treated as such.

      Run the following command to create the file:

      • sudo nano app/__init__.py

      Packages in Python allow you to group modules into logical namespaces or hierarchies. This approach enables the code to be broken down into individual and manageable blocks that perform specific functions.

      Next, you will add code to the __init__.py that will create a Flask instance and import the logic from the views.py file, which you will create after saving this file. Add the following code to your new file:

      /var/www/TestApp/__init__.py

      from flask import Flask
      app = Flask(__name__)
      from app import views
      

      Once you've added that code, save and close the file.

      With the __init__.py file created, you're ready to create the views.py file in your app directory. This file will contain most of your application logic.

      Next, add the code to your views.py file. This code will return the hello world! string to users who visit your web page:

      /var/www/TestApp/app/views.py

      from app import app
      
      @app.route('/')
      def home():
         return "hello world!"
      

      The @app.route line above the function is called a decorator. Decorators modify the function that follows it. In this case, the decorator tells Flask which URL will trigger the home() function. The hello world text returned by the home function will be displayed to the user on the browser.

      With the views.py file in place, you're ready to create the uwsgi.ini file. This file will contain the uWSGI configurations for our application. uWSGI is a deployment option for Nginx that is both a protocol and an application server; the application server can serve uWSGI, FastCGI, and HTTP protocols.

      To create this file, run the following command:

      Next, add the following content to your file to configure the uWSGI server:

      /var/www/TestApp/uwsgi.ini

      [uwsgi]
      module = main
      callable = app
      master = true
      

      This code defines the module that the Flask application will be served from. In this case, this is the main.py file, referenced here as main. The callable option instructs uWSGI to use the app instance exported by the main application. The master option allows your application to keep running, so there is little downtime even when reloading the entire application.

      Next, create the main.py file, which is the entry point to the application. The entry point instructs uWSGI on how to interact with the application.

      Next, copy and paste the following into the file. This imports the Flask instance named app from the application package that was previously created.

      /var/www/TestApp/main.py

      from app import app
      

      Finally, create a requirements.txt file to specify the dependencies that the pip package manager will install to your Docker deployment:

      • sudo nano requirements.txt

      Add the following line to add Flask as a dependency:

      /var/www/TestApp/app/requirements.txt

      Flask==1.0.2
      

      This specifies the version of Flask to be installed. At the time of writing this tutorial, 1.0.2 is the latest Flask version. You can check for updates at the official website for Flask.

      Save and close the file. You have successfully set up your Flask application and are ready to set up Docker.

      Step 2 — Setting Up Docker

      In this step you will create two files, Dockerfile and start.sh, to create your Docker deployment. The Dockerfile is a text document that contains the commands used to assemble the image. The start.sh file is a shell script that will build an image and create a container from the Dockerfile.

      First, create the Dockerfile.

      Next, add your desired configuration to the Dockerfile. These commands specify how the image will be built, and what extra requirements will be included.

      /var/www/TestApp/Dockerfile

      FROM tiangolo/uwsgi-nginx-flask:python3.6-alpine3.7
      RUN apk --update add bash nano
      ENV STATIC_URL /static
      ENV STATIC_PATH /var/www/app/static
      COPY ./requirements.txt /var/www/requirements.txt
      RUN pip install -r /var/www/requirements.txt
      

      In this example, the Docker image will be built off an existing image, tiangolo/uwsgi-nginx-flask, which you can find on DockerHub. This particular Docker image is a good choice over others because it supports a wide range of Python versions and OS images.

      The first two lines specify the parent image that you'll use to run the application and install the bash command processor and the nano text editor. It also installs the git client for pulling and pushing to version control hosting services such as GitHub, GitLab, and Bitbucket. ENV STATIC_URL /static is an environment variable specific to this Docker image. It defines the static folder where all assets such as images, CSS files, and JavaScript files are served from.

      The last two lines will copy the requirements.txt file into the container so that it can be executed, and then parses the requirements.txt file to install the specified dependencies.

      Save and close the file after adding your configuration.

      With your Dockerfile in place, you're almost ready to write your start.sh script that will build the Docker container. Before writing the start.sh script, first make sure that you have an open port to use in the configuration. To check if a port is free, run the following command:

      • sudo nc localhost 56733 < /dev/null; echo $?

      If the output of the command above is 1, then the port is free and usable. Otherwise, you will need to select a different port to use in your start.sh configuration file.

      Once you've found an open port to use, create the start.sh script:

      The start.sh script is a shell script that will build an image from the Dockerfile and create a container from the resulting Docker image. Add your configuration to the new file:

      /var/www/TestApp/start.sh

      #!/bin/bash
      app="docker.test"
      docker build -t ${app} .
      docker run -d -p 56733:80 
        --name=${app} 
        -v $PWD:/app ${app}
      

      The first line is called a shebang. It specifies that this is a bash file and will be executed as commands. The next line specifies the name you want to give the image and container and saves as a variable named app. The next line instructs Docker to build an image from your Dockerfile located in the current directory. This will create an image called docker.test in this example.

      The last three lines create a new container named docker.test that is exposed at port 56733. Finally, it links the present directory to the /var/www directory of the container.

      You use the -d flag to start a container in daemon mode, or as a background process. You include the -p flag to bind a port on the server to a particular port on the Docker container. In this case, you are binding port 56733 to port 80 on the Docker container. The -v flag specifies a Docker volume to mount on the container, and in this case, you are mounting the entire project directory to the /var/www folder on the Docker container.

      Execute the start.sh script to create the Docker image and build a container from the resulting image:

      Once the script finishes running, use the following command to list all running containers:

      You will receive output that shows the containers:

      Output

      CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 58b05508f4dd docker.test "/entrypoint.sh /sta…" 12 seconds ago Up 3 seconds 443/tcp, 0.0.0.0:56733->80/tcp docker.test

      You will find that the docker.test container is running. Now that it is running, visit the IP address at the specified port in your browser: http://ip-address:56733

      You'll see a page similar to the following:

      the home page

      In this step you have successfully deployed your Flask application on Docker. Next, you will use templates to display content to users.

      Step 3 — Serving Template Files

      Templates are files that display static and dynamic content to users who visit your application. In this step, you will create a HTML template to create a home page for the application.

      Start by creating a home.html file in the app/templates directory:

      • sudo nano app/templates/home.html

      Add the code for your template. This code will create an HTML5 page that contains a title and some text.

      /var/www/TestApp/app/templates/home.html

      
      <!doctype html>
      
      <html lang="en-us">   
        <head>
          <meta charset="utf-8">
          <meta http-equiv="x-ua-compatible" content="ie=edge">
          <title>Welcome home</title>
        </head>
      
        <body>
          <h1>Home Page</h1>
          <p>This is the home page of our application.</p>
        </body> 
      </html>
      

      Save and close the file once you've added your template.

      Next, modify the app/views.py file to serve the newly created file:

      First, add the following line at the beginning of your file to import the render_template method from Flask. This method parses an HTML file to render a web page to the user.

      /var/www/TestApp/app/views.py

      from flask import render_template
      ...
      

      At the end of the file, you will also add a new route to render the template file. This code specifies that users are served the contents of the home.html file whenever they visit the /template route on your application.

      /var/www/TestApp/app/views.py

      ...
      
      @app.route('/template')
      def template():
          return render_template('home.html')
      

      The updated app/views.py file will look like this:

      /var/www/TestApp/app/views.py

      from flask import render_template
      from app import app 
      
      @app.route('/')
      def home():
          return "Hello world!"
      
      @app.route('/template')
      def template():
          return render_template('home.html')
      

      Save and close the file when done.

      In order for these changes to take effect, you will need to stop and restart the Docker containers. Run the following command to rebuild the container:

      • sudo docker stop docker.test && sudo docker start docker.test

      Visit your application at http://your-ip-address:56733/template to see the new template being served.

      homepage

      In this you've created a Docker template file to serve visitors on your application. In the next step you will see how the changes you make to your application can take effect without having to restart the Docker container.

      Step 4 — Updating the Application

      Sometimes you will need to make changes to the application, whether it is installing new requirements, updating the Docker container, or HTML and logic changes. In this section, you will configure touch-reload to make these changes without needing to restart the Docker container.

      Python autoreloading watches the entire file system for changes and refreshes the application when it detects a change. Autoreloading is discouraged in production because it can become resource intensive very quickly. In this step, you will use touch-reload to watch for changes to a particular file and reload when the file is updated or replaced.

      To implement this, start by opening your uwsgi.ini file:

      Next, add the highlighted line to the end of the file:

      /var/www/TestApp/uwsgi.ini

      module = main
      callable = app
      master = true
      touch-reload = /app/uwsgi.ini
      

      This specifies a file that will be modified to trigger an entire application reload. Once you've made the changes, save and close the file.

      To demonstrate this, make a small change to your application. Start by opening your app/views.py file:

      Replace the string returned by the home function:

      /var/www/TestApp/app/views.py

      • from flask import render_template
      • from app import app
      • @app.route('/')
      • def home():
      • return "<b>There has been a change</b>"
      • @app.route('/template')
      • def template():
      • return render_template('home.html')

      Save and close the file after you've made a change.

      Next, if you open your application’s homepage at http://ip-address:56733, you will notice that the changes are not reflected. This is because the condition for reload is a change to the uwsgi.ini file. To reload the application, use touch to activate the condition:

      Reload the application homepage in your browser again. You will find that the application has incorporated the changes:

      Homepage Updated

      In this step, you set up a touch-reload condition to update your application after making changes.

      Conclusion

      In this tutorial, you created and deployed a Flask application to a Docker container. You also configured touch-reload to refresh your application without needing to restart the container.

      With your new application on Docker, you can now scale with ease. To learn more about using Docker, check out their official documentation.



      Source link