One place for hosting & domains

      Archives hassan latif

      How To Build a Media Processing API in Node.js With Express and FFmpeg.wasm


      The author selected the Electronic Frontier Foundation to receive a donation as part of the Write for DOnations program.

      Introduction

      Handling media assets is becoming a common requirement of modern back-end services. Using dedicated, cloud-based solutions may help when you’re dealing with massive scale or performing expensive operations, such as video transcoding. However, the extra cost and added complexity may be hard to justify when all you need is to extract a thumbnail from a video or check that user-generated content is in the correct format. Particularly at a smaller scale, it makes sense to add media processing capability directly to your Node.js API.

      In this guide, you will build a media API in Node.js with Express and ffmpeg.wasm — a WebAssembly port of the popular media processing tool. You’ll build an endpoint that extracts a thumbnail from a video as an example. You can use the same techniques to add other features supported by FFmpeg to your API.

      When you’re finished, you will have a good grasp on handling binary data in Express and processing them with ffmpeg.wasm. You’ll also handle requests made to your API that cannot be processed in parallel.

      Prerequisites

      To complete this tutorial, you will need:

      This tutorial was verified with Node v16.11.0, npm v7.15.1, express v4.17.1, and ffmpeg.wasm v0.10.1.

      Step 1 — Setting Up the Project and Creating a Basic Express Server

      In this step, you will create a project directory, initialize Node.js and install ffmpeg, and set up a basic Express server.

      Start by opening the terminal and creating a new directory for the project:

      Navigate to the new directory:

      Use npm init to create a new package.json file. The -y parameter indicates that you’re happy with the default settings for the project.

      Finally, use npm install to install the packages required to build the API. The --save flag indicates that you wish to save those as dependencies in the package.json file.

      • npm install --save @ffmpeg/ffmpeg @ffmpeg/core express cors multer p-queue

      Now that you have installed ffmpeg, you’ll set up a web server that responds to requests using Express.

      First, open a new file called server.mjs with nano or your editor of choice:

      The code in this file will register the cors middleware which will permit requests made from websites with a different origin. At the top of the file, import the express and cors dependencies:

      server.mjs

      import express from 'express';
      import cors from 'cors';
      

      Then, create an Express app and start the server on the port :3000 by adding the following code below the import statements:

      server.mjs

      ...
      const app = express();
      const port = 3000;
      
      app.use(cors());
      
      app.listen(port, () => {
          console.log(`[info] ffmpeg-api listening at http://localhost:${port}`)
      });
      

      You can start the server by running the following command:

      You’ll see the following output:

      Output

      [info] ffmpeg-api listening at http://localhost:3000

      When you try loading http://localhost:3000 in your browser, you’ll see Cannot GET /. This is Express telling you it is listening for requests.

      With your Express server now set up, you’ll create a client to upload the video and make requests to your Express server.

       Step 2 — Creating a Client and Testing the Server

      In this section, you’ll create a web page that will let you select a file and upload it to the API for processing.

      Start by opening a new file called client.html:

      In your client.html file, create a file input and a Create Thumbnail button. Below, add an empty <div> element to display errors and an image that will show the thumbnail that the API sends back. At the very end of the <body> tag, load a script called client.js. Your final HTML template should look as follows:

      client.html

      <!DOCTYPE html>
      <html lang="en">
      <head>
          <meta charset="UTF-8">
          <title>Create a Thumbnail from a Video</title>
          <style>
              #thumbnail {
                  max-width: 100%;
              }
          </style>
      </head>
      <body>
          <div>
              <input id="file-input" type="file" />
              <button id="submit">Create Thumbnail</button>
              <div id="error"></div>
              <img id="thumbnail" />
          </div>
          <script src="https://www.digitalocean.com/community/tutorials/client.js"></script>
      </body>
      </html>
      

      Note that each element has a unique id. You’ll need them when referring to the elements from the client.js script. The styling on the #thumbnail element is there to ensure that the image fits on the screen when it loads.

      Save the client.html file and open client.js:

      In your client.js file, start by defining variables that store references to your HTML elements you created:

      client.js

      const fileInput = document.querySelector('#file-input');
      const submitButton = document.querySelector('#submit');
      const thumbnailPreview = document.querySelector('#thumbnail');
      const errorDiv = document.querySelector('#error');
      

      Then, attach a click event listener to the submitButton variable to check whether you’ve selected a file:

      client.js

      ...
      submitButton.addEventListener('click', async () => {
          const { files } = fileInput;
      }
      

      Next, create a function showError() that will output an error message when a file is not selected. Add the showError() function above your event listener:

      client.js

      const fileInput = document.querySelector('#file-input');
      const submitButton = document.querySelector('#submit');
      const thumbnailPreview = document.querySelector('#thumbnail');
      const errorDiv = document.querySelector('#error');
      
      function showError(msg) {
          errorDiv.innerText = `ERROR: ${msg}`;
      }
      
      submitButton.addEventListener('click', async () => {
      ...
      

      Now, you will build a function createThumbnail() that will make a request to the API, send the video, and receive a thumbnail in response. At the top of your client.js file, define a new constant with the URL to a /thumbnail endpoint:

      const API_ENDPOINT = 'http://localhost:3000/thumbnail';
      
      const fileInput = document.querySelector('#file-input');
      const submitButton = document.querySelector('#submit');
      const thumbnailPreview = document.querySelector('#thumbnail');
      const errorDiv = document.querySelector('#error');
      ...
      

      You will define and use the /thumbnail endpoint in your Express server.

      Next, add the createThumbnail() function below your showError() function:

      client.js

      ...
      function showError(msg) {
          errorDiv.innerText = `ERROR: ${msg}`;
      }
      
      async function createThumbnail(video) {
      
      }
      ...
      

      Web APIs frequently use JSON to transfer structured data from and to the client. To include a video in a JSON, you would have to encode it in base64, which would increase its size by about 30%. You can avoid this by using multipart requests instead. Multipart requests allow you to transfer structured data including binary files over http without the unnecessary overhead. You can do this using the FormData() constructor function.

      Inside the createThumbnail() function, create an instance of FormData and append the video file to the object. Then make a POST request to the API endpoint using the Fetch API with the FormData() instance as the body. Interpret the response as a binary file (or blob) and convert it to a data URL so that you can assign it to the <img> tag you created earlier.

      Here’s the full implementation of createThumbnail():

      client.js

      ...
      async function createThumbnail(video) {
          const payload = new FormData();
          payload.append('video', video);
      
          const res = await fetch(API_ENDPOINT, {
              method: 'POST',
              body: payload
          });
      
          if (!res.ok) {
              throw new Error('Creating thumbnail failed');
          }
      
          const thumbnailBlob = await res.blob();
          const thumbnail = await blobToDataURL(thumbnailBlob);
      
          return thumbnail;
      }
      ...
      

      You’ll notice createThumbnail() has the function blobToDataURL() in its body. This is a helper function that will convert a blob to a data URL.

      Above your createThumbnail() function, create the function blobDataToURL() that returns a promise:

      client.js

      ...
      async function blobToDataURL(blob) {
          return new Promise((resolve, reject) => {
              const reader = new FileReader();
              reader.onload = () => resolve(reader.result);
              reader.onerror = () => reject(reader.error);
              reader.onabort = () => reject(new Error("Read aborted"));
              reader.readAsDataURL(blob);
          });
      }
      ...
      

      blobToDataURL() uses FileReader to read the contents of the binary file and format it as a data URL.

      With the createThumbnail() and showError() functions now defined, you can use them to finish implementing the event listener:

      client.js

      ...
      submitButton.addEventListener('click', async () => {
          const { files } = fileInput;
      
          if (files.length > 0) {
              const file = files[0];
              try {
                  const thumbnail = await createThumbnail(file);
                  thumbnailPreview.src = thumbnail;
              } catch(error) {
                  showError(error);
              }
          } else {
              showError('Please select a file');
          }
      });
      

      When a user clicks on the button, the event listener will pass the file to the createThumbnail() function. If successful, it will assign the thumbnail to the <img> element you created earlier. In case the user doesn’t select a file or the request fails, it will call the showError() function to display an error.

      At this point, your client.js file will look like the following:

      client.js

      const API_ENDPOINT = 'http://localhost:3000/thumbnail';
      
      const fileInput = document.querySelector('#file-input');
      const submitButton = document.querySelector('#submit');
      const thumbnailPreview = document.querySelector('#thumbnail');
      const errorDiv = document.querySelector('#error');
      
      function showError(msg) {
          errorDiv.innerText = `ERROR: ${msg}`;
      }
      
      async function blobToDataURL(blob) {
          return new Promise((resolve, reject) => {
              const reader = new FileReader();
              reader.onload = () => resolve(reader.result);
              reader.onerror = () => reject(reader.error);
              reader.onabort = () => reject(new Error("Read aborted"));
              reader.readAsDataURL(blob);
          });
      }
      
      async function createThumbnail(video) {
          const payload = new FormData();
          payload.append('video', video);
      
          const res = await fetch(API_ENDPOINT, {
              method: 'POST',
              body: payload
          });
      
          if (!res.ok) {
              throw new Error('Creating thumbnail failed');
          }
      
          const thumbnailBlob = await res.blob();
          const thumbnail = await blobToDataURL(thumbnailBlob);
      
          return thumbnail;
      }
      
      submitButton.addEventListener('click', async () => {
          const { files } = fileInput;
      
          if (files.length > 0) {
              const file = files[0];
      
              try {
                  const thumbnail = await createThumbnail(file);
                  thumbnailPreview.src = thumbnail;
              } catch(error) {
                  showError(error);
              }
          } else {
              showError('Please select a file');
          }
      });
      

      Start the server again by running:

      With your client now set up, uploading the video file here will result in receiving an error message. This is because the /thumbnail endpoint is not built yet. In the next step, you’ll create the /thumbnail endpoint in Express to accept the video file and create the thumbnail.

       Step 3 — Setting Up an Endpoint to Accept Binary Data

      In this step, you will set up a POST request for the /thumbnail endpoint and use middleware to accept multipart requests.

      Open server.mjs in an editor:

      Then, import multer at the top of the file:

      server.mjs

      import express from 'express';
      import cors from 'cors';
      import multer from 'multer';
      ...
      

      Multer is a middleware that processes incoming multipart/form-data requests before passing them to your endpoint handler. It extracts fields and files from the body and makes them available as an array on the request object in Express. You can configure where to store the uploaded files and set limits on file size and format.

      After importing it, initialize the multer middleware with the following options:

      server.mjs

      ...
      const app = express();
      const port = 3000;
      
      const upload = multer({
          storage: multer.memoryStorage(),
          limits: { fileSize: 100 * 1024 * 1024 }
      });
      
      app.use(cors());
      ...
      

      The storage option lets you choose where to store the incoming files. Calling multer.memoryStorage() will initialize a storage engine that keeps files in Buffer objects in memory as opposed to writing them to disk. The limits option lets you define various limits on what files will be accepted. Set the fileSize limit to 100MB or a different number that matches your needs and the amount of memory available on your server. This will prevent your API from crashing when the input file is too big.

      Note: Due to the limitations of WebAssembly, ffmpeg.wasm cannot handle input files over 2GB in size.

      Next, set up the POST /thumbnail endpoint itself:

      server.mjs

      ...
      app.use(cors());
      
      app.post('/thumbnail', upload.single('video'), async (req, res) => {
          const videoData = req.file.buffer;
      
          res.sendStatus(200);
      });
      
      app.listen(port, () => {
          console.log(`[info] ffmpeg-api listening at http://localhost:${port}`)
      });
      

      The upload.single('video') call will set up a middleware for that endpoint only that will parse the body of a multipart request that includes a single file. The first parameter is the field name. It must match the one you gave to FormData when creating the request in client.js. In this case, it’s video. multer will then attach the parsed file to the req parameter. The content of the file will be under req.file.buffer.

      At this point, the endpoint doesn’t do anything with the data it receives. It acknowledges the request by sending an empty 200 response. In the next step, you’ll replace that with the code that extracts a thumbnail from the video data received.

      In this step, you’ll use ffmpeg.wasm to extract a thumbnail from the video file received by the POST /thumbnail endpoint.

      ffmpeg.wasm is a pure WebAssembly and JavaScript port of FFmpeg. Its main goal is to allow running FFmpeg directly in the browser. However, because Node.js is built on top of V8 — Chrome’s JavaScript engine — you can use the library on the server too.

      The benefit of using a native port of FFmpeg over a wrapper built on top of the ffmpeg command is that if you’re planning to deploy your app with Docker, you don’t have to build a custom image that includes both FFmpeg and Node.js. This will save you time and reduce the maintenance burden of your service.

      Add the following import to the top of server.mjs:

      server.mjs

      import express from 'express';
      import cors from 'cors';
      import multer from 'multer';
      import { createFFmpeg } from '@ffmpeg/ffmpeg';
      ...
      

      Then, create an instance of ffmpeg.wasm and start loading the core:

      server.mjs

      ...
      import { createFFmpeg } from '@ffmpeg/ffmpeg';
      
      const ffmpegInstance = createFFmpeg({ log: true });
      let ffmpegLoadingPromise = ffmpegInstance.load();
      
      const app = express();
      ...
      

      The ffmpegInstance variable holds a reference to the library. Calling ffmpegInstance.load() starts loading the core into memory asynchronously and returns a promise. Store the promise in the ffmpegLoadingPromise variable so that you can check whether the core has loaded.

      Next, define the following helper function that will use fmpegLoadingPromise to wait for the core to load in case the first request arrives before it’s ready:

      server.mjs

      ...
      let ffmpegLoadingPromise = ffmpegInstance.load();
      
      async function getFFmpeg() {
          if (ffmpegLoadingPromise) {
              await ffmpegLoadingPromise;
              ffmpegLoadingPromise = undefined;
          }
      
          return ffmpegInstance;
      }
      
      const app = express();
      ...
      

      The getFFmpeg() function returns a reference to the library stored in the ffmpegInstance variable. Before returning it, it checks whether the library has finished loading. If not, it will wait until ffmpegLoadingPromise resolves. In case the first request to your POST /thumbnail endpoint arrives before ffmpegInstance is ready to use, your API will wait and resolve it when it can rather than rejecting it.

      Now, implement the POST /thumbnail endpoint handler. Replace res.sendStatus(200); at the end of the end of the function with a call to getFFmpeg to get a reference to ffmpeg.wasm when it’s ready:

      server.mjs

      ...
      app.post('/thumbnail', upload.single('video'), async (req, res) => {
          const videoData = req.file.buffer;
      
          const ffmpeg = await getFFmpeg();
      });
      ...
      

      ffmpeg.wasm works on top of an in-memory file system. You can read and write to it using ffmpeg.FS. When running FFmpeg operations, you will pass virtual file names to the ffmpeg.run function as an argument the same way as you would when working with the CLI tool. Any output files created by FFmpeg will be written to the file system for you to retrieve.

      In this case, the input file is a video. The output file will be a single PNG image. Define the following variables:

      server.mjs

      ...
          const ffmpeg = await getFFmpeg();
      
          const inputFileName = `input-video`;
          const outputFileName = `output-image.png`;
          let outputData = null;
      });
      ...
      

      The file names will be used on the virtual file system. outputData is where you’ll store the thumbnail when it’s ready.

      Call ffmpeg.FS() to write the video data to the in-memory file system:

      server.mjs

      ...
          let outputData = null;
      
          ffmpeg.FS('writeFile', inputFileName, videoData);
      });
      ...
      

      Then, run the FFmpeg operation:

      server.mjs

      ...
          ffmpeg.FS('writeFile', inputFileName, videoData);
      
          await ffmpeg.run(
              '-ss', '00:00:01.000',
              '-i', inputFileName,
              '-frames:v', '1',
              outputFileName
          );
      });
      ...
      

      The -i parameter specifies the input file. -ss seeks to the specified time (in this case, 1 second from the beginning of the video). -frames:v limits the number of frames that will be written to the output (a single frame in this scenario). outputFileName at the end indicates where will FFmpeg write the output.

      After FFmpeg exits, use ffmpeg.FS() to read the data from the file system and delete both the input and output files to free up memory:

      server.mjs

      ...
          await ffmpeg.run(
              '-ss', '00:00:01.000',
              '-i', inputFileName,
              '-frames:v', '1',
              outputFileName
          );
      
          outputData = ffmpeg.FS('readFile', outputFileName);
          ffmpeg.FS('unlink', inputFileName);
          ffmpeg.FS('unlink', outputFileName);
      });
      ...
      

      Finally, dispatch the output data in the body of the response:

      server.mjs

      ...
          ffmpeg.FS('unlink', outputFileName);
      
          res.writeHead(200, {
              'Content-Type': 'image/png',
              'Content-Disposition': `attachment;filename=${outputFileName}`,
              'Content-Length': outputData.length
          });
          res.end(Buffer.from(outputData, 'binary'));
      });
      ...
      

      Calling res.writeHead() dispatches the response head. The second parameter includes custom http headers) with information about the data in the body of the request that will follow. The res.end() function sends the data from its first argument as the body of the request and finalizes the request. The outputData variable is a raw array of bytes as returned by ffmpeg.FS(). Passing it to Buffer.from() initializes a Buffer to ensure the binary data will be handled correctly by res.end().

      At this point, your POST /thumbnail endpoint implementation should look like this:

      server.mjs

      ...
      app.post('/thumbnail', upload.single('video'), async (req, res) => {
          const videoData = req.file.buffer;
      
          const ffmpeg = await getFFmpeg();
      
          const inputFileName = `input-video`;
          const outputFileName = `output-image.png`;
          let outputData = null;
      
          ffmpeg.FS('writeFile', inputFileName, videoData);
      
          await ffmpeg.run(
              '-ss', '00:00:01.000',
              '-i', inputFileName,
              '-frames:v', '1',
              outputFileName
          );
      
          outputData = ffmpeg.FS('readFile', outputFileName);
          ffmpeg.FS('unlink', inputFileName);
          ffmpeg.FS('unlink', outputFileName);
      
          res.writeHead(200, {
              'Content-Type': 'image/png',
              'Content-Disposition': `attachment;filename=${outputFileName}`,
              'Content-Length': outputData.length
          });
          res.end(Buffer.from(outputData, 'binary'));
      });
      ...
      

      Aside from the 100MB file limit for uploads, there’s no input validation or error handling. When ffmpeg.wasm fails to process a file, reading the output from the virtual file system will fail and prevent the response from being sent. For the purposes of this tutorial, wrap the implementation of the endpoint in a try-catch block to handle that scenario:

      server.mjs

      ...
      app.post('/thumbnail', upload.single('video'), async (req, res) => {
          try {
              const videoData = req.file.buffer;
      
              const ffmpeg = await getFFmpeg();
      
              const inputFileName = `input-video`;
              const outputFileName = `output-image.png`;
              let outputData = null;
      
              ffmpeg.FS('writeFile', inputFileName, videoData);
      
              await ffmpeg.run(
                  '-ss', '00:00:01.000',
                  '-i', inputFileName,
                  '-frames:v', '1',
                  outputFileName
              );
      
              outputData = ffmpeg.FS('readFile', outputFileName);
              ffmpeg.FS('unlink', inputFileName);
              ffmpeg.FS('unlink', outputFileName);
      
              res.writeHead(200, {
                  'Content-Type': 'image/png',
                  'Content-Disposition': `attachment;filename=${outputFileName}`,
                  'Content-Length': outputData.length
              });
              res.end(Buffer.from(outputData, 'binary'));
          } catch(error) {
              console.error(error);
              res.sendStatus(500);
          }
      ...
      });
      

      Secondly, ffmpeg.wasm cannot handle two requests in parallel. You can try this yourself by launching the server:

      • node --experimental-wasm-threads server.mjs

      Note the flag required for ffmpeg.wasm to work. The library depends on WebAssembly threads and bulk memory operations. These have been in V8/Chrome since 2019. However, as of Node.js v16.11.0, WebAssembly threads remain behind a flag in case there might be changes before the proposal is finalised. Bulk memory operations also require a flag in older versions of Node. If you’re running Node.js 15 or lower, add --experimental-wasm-bulk-memory as well.

      The output of the command will look like this:

      Output

      [info] use ffmpeg.wasm v0.10.1 [info] load ffmpeg-core [info] loading ffmpeg-core [info] fetch ffmpeg.wasm-core script from @ffmpeg/core [info] ffmpeg-api listening at http://localhost:3000 [info] ffmpeg-core loaded

      Open client.html in a web browser and select a video file. When you click the Create Thumbnail button, you should see the thumbnail appear on the page. Behind the scenes, the site uploads the video to the API, which processes it and responds with the image. However, when you click the button repeatedly in quick succession, the API will handle the first request. The subsequent requests will fail:

      Output

      Error: ffmpeg.wasm can only run one command at a time at Object.run (.../ffmpeg-api/node_modules/@ffmpeg/ffmpeg/src/createFFmpeg.js:126:13) at file://.../ffmpeg-api/server.mjs:54:26 at runMicrotasks (<anonymous>) at processTicksAndRejections (internal/process/task_queues.js:95:5)

      In the next section, you’ll learn how to deal with concurrent requests.

      Step 5 — Handling Concurrent Requests

      Since ffmpeg.wasm can only execute a single operation at a time, you’ll need a way of serializing requests that come in and processing them one at a time. In this scenario, a promise queue is a perfect solution. Instead of starting to process each request right away, it will be queued up and processed when all the requests that arrived before it have been handled.

      Open server.mjs in your preferred editor:

      Import p-queue at the top of server.mjs:

      server.mjs

      import express from 'express';
      import cors from 'cors';
      import { createFFmpeg } from '@ffmpeg/ffmpeg';
      import PQueue from 'p-queue';
      ...
      

      Then, create a new queue at the top of server.mjs file under the variable ffmpegLoadingPromise:

      server.mjs

      ...
      const ffmpegInstance = createFFmpeg({ log: true });
      let ffmpegLoadingPromise = ffmpegInstance.load();
      
      const requestQueue = new PQueue({ concurrency: 1 });
      ...
      

      In the POST /thumbnail endpoint handler, wrap the calls to ffmpeg in a function that will be queued up:

      server.mjs

      ...
      app.post('/thumbnail', upload.single('video'), async (req, res) => {
          try {
              const videoData = req.file.buffer;
      
              const ffmpeg = await getFFmpeg();
      
              const inputFileName = `input-video`;
              const outputFileName = `thumbnail.png`;
              let outputData = null;
      
              await requestQueue.add(async () => {
                  ffmpeg.FS('writeFile', inputFileName, videoData);
      
                  await ffmpeg.run(
                      '-ss', '00:00:01.000',
                      '-i', inputFileName,
                      '-frames:v', '1',
                      outputFileName
                  );
      
                  outputData = ffmpeg.FS('readFile', outputFileName);
                  ffmpeg.FS('unlink', inputFileName);
                  ffmpeg.FS('unlink', outputFileName);
              });
      
              res.writeHead(200, {
                  'Content-Type': 'image/png',
                  'Content-Disposition': `attachment;filename=${outputFileName}`,
                  'Content-Length': outputData.length
              });
              res.end(Buffer.from(outputData, 'binary'));
          } catch(error) {
              console.error(error);
              res.sendStatus(500);
          }
      });
      ...
      

      Every time a new request comes in, it will only start processing when there’s nothing else queued up in front of it. Note that the final sending of the response can happen asynchronously. Once the ffmpeg.wasm operation finishes running, another request can start processing while the response goes out.

      To test that everything works as expected, start up the server again:

      • node --experimental-wasm-threads server.mjs

      Open the client.html file in your browser and try uploading a file.

      A screenshot of client.html with a thumbnail loaded

      With the queue in place, the API will now respond every time. The requests will be handled sequentially in the order in which they arrive.

      Conclusion

      In this article, you built a Node.js service that extracts a thumbnail from a video using ffmpeg.wasm. You learned how to upload binary data from the browser to your Express API using multipart requests and how to process media with FFmpeg in Node.js without relying on external tools or having to write data to disk.

      FFmpeg is an incredibly versatile tool. You can use the knowledge from this tutorial to take advantage of any features that FFmpeg supports and use them in your project. For example, to generate a three-second GIF, change the ffmpeg.run call to this on the POST /thumbnail endpoint:

      server.mjs

      ...
      await ffmpeg.run(
          '-y',
          '-t', '3',
          '-i', inputFileName,
          '-filter_complex', 'fps=5,scale=720:-1:flags=lanczos[x];[x]split[x1][x2];[x1]palettegen[p];[x2][p]paletteuse',
          '-f', 'gif',
          outputFileName
      );
      ...
      

      The library accepts the same parameters as the original ffmpeg CLI tool. You can use the official documentation to find a solution for your use case and test it quickly in the terminal.

      Thanks to ffmpeg.wasm being self-contained, you can dockerize this service using the stock Node.js base images and scale your service up by keeping multiple nodes behind a load balancer. Follow the tutorial How To Build a Node.js Application with Docker to learn more.

      If your use case requires performing more expensive operations, such as transcoding large videos, make sure that you run your service on machines with enough memory to store them. Due to current limitations in WebAssembly, the maximum input file size cannot exceed 2GB, although this might change in the future.

      Additionally, ffmpeg.wasm cannot take advantage of some x86 assembly optimizations from the original FFmpeg codebase. That means some operations can take a long time to finish. If that’s the case, consider whether this is the right solution for your use case. Alternatively, make requests to your API asynchronous. Instead of waiting for the operation to finish, queue it up and respond with a unique ID. Create another endpoint that the clients can query to find out whether the processing ended and the output file is ready. Learn more about the asynchronous request-reply pattern for REST APIs and how to implement it.



      Source link

      How To Test Your Data With Great Expectations


      The author selected the Diversity in Tech Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      In this tutorial, you will set up a local deployment of Great Expectations, an open source data validation and documentation library written in Python. Data validation is crucial to ensuring that the data you process in your pipelines is correct and free of any data quality issues that might occur due to errors such as incorrect inputs or transformation bugs. Great Expectations allows you to establish assertions about your data called Expectations, and validate any data using those Expectations.

      When you’re finished, you’ll be able to connect Great Expectations to your data, create a suite of Expectations, validate a batch of data using those Expectations, and generate a data quality report with the results of your validation.

      Prerequisites

      To complete this tutorial, you will need:

      Step 1 — Installing Great Expectations and Initializing a Great Expectations Project

      In this step, you will install the Great Expectations package in your local Python environment, download the sample data you’ll use in this tutorial, and initialize a Great Expectations project.

      To begin, open a terminal and make sure to activate your virtual Python environment. Install the Great Expectations Python package and command-line tool (CLI) with the following command:

      • pip install great_expectations==0.13.35

      Note: This tutorial was developed for Great Expectations version 0.13.35 and may not be applicable to other versions.

      In order to have access to the example data repository, run the following git command to clone the directory and change into it as your working directory:

      • git clone https://github.com/do-community/great_expectations_tutorial
      • cd great_expectations_tutorial

      The repository only contains one folder called data, which contains two example CSV files with data that you will use in this tutorial. Take a look at the contents of the data directory:

      You’ll see the following output:

      Output

      yellow_tripdata_sample_2019-01.csv yellow_tripdata_sample_2019-02.csv

      Great Expectations works with many different types of data, such as connections to relational databases, Spark dataframes, and various file formats. For the purpose of this tutorial, you will use these CSV files containing a small set of taxi ride data to get started.

      Finally, initialize your directory as a Great Expectations project by running the following command. Make sure to use the --v3-api flag, as this will switch you to using the most recent API of the package:

      • great_expectations --v3-api init

      When asked OK to proceed? [Y/n]:, press ENTER to proceed.

      This will create a folder called great_expectations, which contains the basic configuration for your Great Expectations project, also called the Data Context. You can inspect the contents of the folder:

      You will see the first level of files and subdirectories that were created inside the great_expectations folder:

      Output

      checkpoints great_expectations.yml plugins expectations notebooks uncommitted

      The folders store all the relevant content for your Great Expectations setup. The great_expectations.yml file contains all important configuration information. Feel free to explore the folders and configuration file a little more before moving on to the next step in the tutorial.

      In the next step, you will add a Datasource to point Great Expectations at your data.

      Step 2 — Adding a Datasource

      In this step, you will configure a Datasource in Great Expectations, which allows you to automatically create data assertions called Expectations as well as validate data with the tool.

      While in your project directory, run the following command:

      • great_expectations --v3-api datasource new

      You will see the following output. Enter the options shown when prompted to configure a file-based Datasource for the data directory:

      Output

      What data would you like Great Expectations to connect to? 1. Files on a filesystem (for processing with Pandas or Spark) 2. Relational database (SQL) : 1 What are you processing your files with? 1. Pandas 2. PySpark : 1 Enter the path of the root directory where the data files are stored. If files are on local disk enter a path relative to your current working directory or an absolute path. : data

      After confirming the directory path with ENTER, Great Expectations will open a Jupyter notebook in your web browser, which allows you to complete the configuration of the Datasource and store it to your Data Context. The following screenshot shows the first few cells of the notebook.

      Screenshot of a Jupyter notebook

      The notebook contains several pre-populated cells of Python code to configure your Datasource. You can modify the settings for the Datasource, such as the name, if you like. However, for the purpose of this tutorial, you’ll leave everything as-is and execute all cells using the Cell > Run All menu option. If run successfully, the last cell output will look as follows:

      Output

      [{'data_connectors': {'default_inferred_data_connector_name': {'module_name': 'great_expectations.datasource.data_connector', 'base_directory': '../data', 'class_name': 'InferredAssetFilesystemDataConnector', 'default_regex': {'group_names': ['data_asset_name'], 'pattern': '(.*)'}}, 'default_runtime_data_connector_name': {'module_name': 'great_expectations.datasource.data_connector', 'class_name': 'RuntimeDataConnector', 'batch_identifiers': ['default_identifier_name']}}, 'module_name': 'great_expectations.datasource', 'class_name': 'Datasource', 'execution_engine': {'module_name': 'great_expectations.execution_engine', 'class_name': 'PandasExecutionEngine'}, 'name': 'my_datasource'}]

      This shows that you have added a new Datasource called my_datasource to your Data Context. Feel free to read through the instructions in the notebook to learn more about the different configuration options before moving on to the next step.

      Warning: Before moving forward, close the browser tab with the notebook, return to your terminal, and press CTRL+C to shut down the running notebook server before proceeding.

      You have now successfully set up a Datasource that points at the data directory, which will allow you to access the CSV files in the directory through Great Expectations. In the next step, you will use one of these CSV files in your Datasource to automatically generate Expectations with a profiler.

      Step 3 — Creating an Expectation Suite With an Automated Profiler

      In this step of the tutorial, you will use the built-in Profiler to create a set of Expectations based on some existing data. For this purpose, let’s take a closer look at the sample data that you downloaded:

      • The files yellow_tripdata_sample_2019-01.csv and yellow_tripdata_sample_2019-02.csv contain taxi ride data from January and February 2019, respectively.
      • This tutorial assumes that you know the January data is correct, and that you want to ensure that any subsequent data files match the January data in terms of number or rows, columns, and the distributions of certain column values.

      For this purpose, you will create Expectations (data assertions) based on certain properties of the January data and then, in a later step, use those Expectations to validate the February data. Let’s get started by creating an Expectation Suite, which is a set of Expectations that are grouped together:

      • great_expectations --v3-api suite new

      By selecting the options shown in the output below, you specify that you would like to use a profiler to generate Expectations automatically, using the yellow_tripdata_sample_2019-01.csv data file as an input. Enter the name my_suite as the Expectation Suite name when prompted and press ENTER at the end when asked Would you like to proceed? [Y/n]:

      Output

      Using v3 (Batch Request) API How would you like to create your Expectation Suite? 1. Manually, without interacting with a sample batch of data (default) 2. Interactively, with a sample batch of data 3. Automatically, using a profiler : 3 A batch of data is required to edit the suite - let's help you to specify it. Which data asset (accessible by data connector "my_datasource_example_data_connector") would you like to use? 1. yellow_tripdata_sample_2019-01.csv 2. yellow_tripdata_sample_2019-02.csv : 1 Name the new Expectation Suite [yellow_tripdata_sample_2019-01.csv.warning]: my_suite When you run this notebook, Great Expectations will store these expectations in a new Expectation Suite "my_suite" here: <path_to_project>/great_expectations_tutorial/great_expectations/expectations/my_suite.json Would you like to proceed? [Y/n]: <press ENTER>

      This will open another Jupyter notebook that lets you complete the configuration of your Expectation Suite. The notebook contains a fair amount of code to configure the built-in profiler, which looks at the CSV file you selected and creates certain types of Expectations for each column in the file based on what it finds in the data.

      Scroll down to the second code cell in the notebook, which contains a list of ignored_columns. By default, the profiler will ignore all columns, so let’s comment out some of them to make sure the profiler creates Expectations for them. Modify the code so it looks like this:

      ignored_columns = [
      #     "vendor_id"
      # ,    "pickup_datetime"
      # ,    "dropoff_datetime"
      # ,    "passenger_count"
          "trip_distance"
      ,    "rate_code_id"
      ,    "store_and_fwd_flag"
      ,    "pickup_location_id"
      ,    "dropoff_location_id"
      ,    "payment_type"
      ,    "fare_amount"
      ,    "extra"
      ,    "mta_tax"
      ,    "tip_amount"
      ,    "tolls_amount"
      ,    "improvement_surcharge"
      ,    "total_amount"
      ,    "congestion_surcharge"
      ,]
      

      Make sure to remove the comma before "trip_distance". By commenting out the columns vendor_id, pickup_datetime, dropoff_datetime, and passenger_count, you are telling the profiler to generate Expectations for those columns. In addition, the profiler will also generate table-level Expectations, such as the number and names of columns in your data, and the number of rows. Once again, execute all cells in the notebook by using the Cell > Run All menu option.

      When executing all cells in this notebook, two things happen:

      1. The code creates an Expectation Suite using the automated profiler and the yellow_tripdata_sample_2019-01.csv file you told it to use.
      2. The last cell in the notebook is also configured to run validation and open a new browser window with Data Docs, which is a data quality report.

      In the next step, you will take a closer look at the Data Docs that were opened in the new browser window.

      Step 4 — Exploring Data Docs

      In this step of the tutorial, you will inspect the Data Docs that Great Expectations generated and learn how to interpret the different pieces of information. Go to the browser window that just opened and take a look at the page, shown in the screenshot below.

      Screenshot of Data Docs

      At the top of the page, you will see a box titled Overview, which contains some information about the validation you just ran using your newly created Expectation Suite my_suite. It will tell you Status: Succeeded and show some basic statistics about how many Expectations were run. If you scroll further down, you will see a section titled Table-Level Expectations. It contains two rows of Expectations, showing the Status, Expectation, and Observed Value for each row. Below the table Expectations, you will see the column-level Expectations for each of the columns you commented out in the notebook.

      Let’s focus on one specific Expectation: The passenger_count column has an Expectation stating “values must belong to this set: 1 2 3 4 5 6.” which is marked with a green checkmark and has an Observed Value of “0% unexpected”. This is telling you that the profiler looked at the values in the passenger_count column in the January CSV file and detected only the values 1 through 6, meaning that all taxi rides had between 1 and 6 passengers. Great Expectations then created an Expectation for this fact. The last cell in the notebook then triggered validation of the January CSV file and it found no unexpected values. This is spuriously true, since the same data that was used to create the Expectation was also the data used for validation.

      In this step, you reviewed the Data Docs and observed the passenger_count column for its Expectation. In the next step, you’ll see how you can validate a different batch of data.

      Step 5 — Creating a Checkpoint and Running Validation

      In the final step of this tutorial, you will create a new Checkpoint, which bundles an Expectation Suite and a batch of data to execute validation of that data. After creating the Checkpoint, you will then run it to validate the February taxi data CSV file and see whether the file passed the Expectations you previously created. To begin, return to your terminal and stop the Jupyter notebook by pressing CTRL+C if it is still running. The following command will start the workflow to create a new Checkpoint called my_checkpoint:

      • great_expectations --v3-api checkpoint new my_checkpoint

      This will open a Jupyter notebook with some pre-populated code to configure the Checkpoint. The second code cell in the notebook will have a random data_asset_name pre-populated from your existing Datasource, which will be one of the two CSV files in the data directory you’ve seen earlier. Ensure that the data_asset_name is yellow_tripdata_sample_2019-02.csv and modify the code if needed to use the correct filename.

      my_checkpoint_name = "my_checkpoint" # This was populated from your CLI command.
      
      yaml_config = f"""
      name: {my_checkpoint_name}
      config_version: 1.0
      class_name: SimpleCheckpoint
      run_name_template: "%Y%m%d-%H%M%S-my-run-name-template"
      validations:
        - batch_request:
            datasource_name: my_datasource
            data_connector_name: default_inferred_data_connector_name
            data_asset_name: yellow_tripdata_sample_2019-02.csv
            data_connector_query:
              index: -1
          expectation_suite_name: my_suite
      """
      print(yaml_config)
      """
      

      This configuration snippet configures a new Checkpoint, which reads the data asset yellow_tripdata_sample_2019-02.csv, i.e., your February CSV file, and validates it using the Expectation Suite my_suite. Confirm that you modified the code correctly, then execute all cells in the notebook. This will save the new Checkpoint to your Data Context.

      Finally, in order to run this new Checkpoint and validate the February data, scroll down to the last cell in the notebook. Uncomment the code in the cell to look as follows:

      context.run_checkpoint(checkpoint_name=my_checkpoint_name)
      context.open_data_docs()
      

      Select the cell and run it using the Cell > Run Cells menu option or the SHIFT+ENTER keyboard shortcut. This will open Data Docs in a new browser tab.

      On the Validation Results overview page, click on the topmost run to navigate to the Validation Result details page. The Validation Result details page will look very similar to the page you saw in the previous step, but it will now show that the Expectation Suite failed, validating the new CSV file. Scroll through the page to see which Expectations have a red X next to them, marking them as failed.

      Find the Expectation on the passenger_count column you looked at in the previous step: “values must belong to this set: 1 2 3 4 5 6”. You will notice that it now shows up as failed and highlights that 1579 unexpected values found. ≈15.79% of 10000 total rows. The row also displays a sample of the unexpected values that were found in the column, namely the value 0. This means that the February taxi ride data suddenly introduced the unexpected value 0 as in the passenger_counts column, which seems like a potential data bug. By running the Checkpoint, you validated the new data with your Expectation Suite and detected this issue.

      Note that each time you execute the run_checkpoint method in the last notebook cell, you kick off another validation run. In a production data pipeline environment, you would call the run_checkpoint command outside of a notebook whenever you’re processing a new batch of data to ensure that the new data passes all validations.

      Conclusion

      In this article you created a first local deployment of the Great Expectations framework for data validation. You initialized a Great Expectations Data Context, created a new file-based Datasource, and automatically generated an Expectation Suite using the built-in profiler. You then created a Checkpoint to run validation against a new batch of data, and inspected the Data Docs to view the validation results.

      This tutorial only taught you the basics of Great Expectations. The package contains more options for configuring Datasources to connect to other types of data, for example relational databases. It also comes with a powerful mechanism to automatically recognize new batches of data based on pattern-matching in the tablename or filename, which allows you to only configure a Checkpoint once to validate any future data inputs. You can learn more about Great Expectations in the official documentation.



      Source link

      How to Dynamically Import JavaScript with Import Maps


      The author selected Creative Commons to receive a donation as part of the Write for DOnations program.

      Introduction

      External libraries can add complexity to a new JavaScript project. To be able to install and use external code libraries, you’ll need a build tool that can parse the code and bundle the libraries that you import into a final format. After the build is set up, you can add and integrate new code with ease, but there are still some problems.

      For example, you may need a library in just one part of your application, a part of the application most users may never need, like an admin page. But by default most build systems will bundle all the code into one large file. The user will load the code regardless of whether they ever need to execute it. The build systems are flexible enough that they can be configured to load code as needed, but the process takes some work.

      Build tools are an important part of the development experience, but a spec called import maps will allow you to both import external code into your project without a build tool and it will only load the code when it is needed at runtime. Import maps won’t completely replace build tools that perform many other valuable actions like building style sheets and handling images, but they will allow you to bootstrap new JavaScript applications quickly and easily using only the native browser functionality.

      In this tutorial, you’ll use import maps and JavaScript modules to import code without build tools. You’ll create a basic application that will display a message and you’ll create an import map that will tell your browser where to locate external code. Next, you’ll integrate the imported code into your JavaScript and will use the third-party code without any need to download the code locally or run it through a build step. Finally, you’ll learn about current tools that implement many aspects of import maps and work on all modern browsers.

      Prerequisites

      Step 1 — Creating an HTML Page and Inserting JavaScript

      In this step, you will create an HTML page, use JavaScript for dynamic activity, and start a local development server to track your changes.

      To start, in a new directory, create a blank HTML document.

      Open a file called index.html in a text editor:

      Inside of the file, add a short, blank HTML page:

      index.html

      <!DOCTYPE html>
      <html lang="en-US">
        <head>
          <meta charset="utf-8">
          <meta http-equiv="X-UA-Compatible" content="IE=edge">
          <title>Hello World</title>
          <meta name="viewport" content="width=device-width, initial-scale=1">
        </head>
        <body>
        </body>
      </html>
      

      This document has a few standard <meta> tags and an empty <body> element.

      Next add a <script> tag. The src attribute for the script tag will be a new JavaScript file you are about to create called hello.js:

      index.html

      <!DOCTYPE html>
      <html lang="en-US">
      <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <title>Hello World</title>
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <script defer src="https://www.digitalocean.com/community/tutorials/./hello.js"></script>
      </head>
      <body>
      </body>
      

      Notice that you are adding a defer attribute to the <script> tag. This will delay execution of the script tag until after the document is parsed. If you don’t defer, you may receive an error that says body is not found when you try to add to the element.

      Next, create a JavaScript file named hello.js in the same directory asindex.html:

      Inside of the file, write some JavaScript to create a new text element with the text "Hello, World":

      hello.js

      const el = document.createElement('h1');
      const words = "Hello, World!"
      const text = document.createTextNode(words);
      el.appendChild(text);
      
      document.body.appendChild(el);
      

      Now that you have your script, you can open the index.html file in a browser. In the same directory as your index.html file, run npx serve. This will run the serve package locally without downloading into your node_modules. The serve package runs a simple webserver that will serve your code locally.

      npx serve
      

      The command will ask you if you want to install a package. Type y to agree:

      Need to install the following packages:
        serve
      Ok to proceed? (y) y
      

      When you run the command you will see some output like this:

      npx: installed 88 in 15.187s
      
         ┌────────────────────────────────────────┐
         │                                        │
         │   Serving!                             │
         │                                        │
         │   Local:  http://localhost:5000        │
         │                                        │
         │   Copied local address to clipboard!   │
         │                                        │
         └────────────────────────────────────────┘
      
      

      When you open your web browser to http://localhost:5000, you’ll see your code. You can either leave the server running in a separate tab or window or close it with CTRL+C after previewing your code.

      Hello, World in a browser

      Now you are displaying a basic page in your browser, but you are not yet able to take advantage of third-party code and JavaScript packages. In the next step, you’ll dynamically import code and import the functions into your script without a build tool.

      Step 2 — Writing a Hello World Script Using ES6 Modules

      In this step, you’ll write code that uses external packages. You’ll modify your code to import JavaScript code using ES imports. Finally, you’ll load the code in your browser using the module type so the browser will know to dynamically load the code.

      To begin, open up hello.js:

      You are going to import some code from lodash to dynamically change your text.

      Inside of the file, change the text from Hello World to all lower case: hello world. Then at the top of the file, import the startCase function from lodash using the standard ES6 import syntax:

      hello.js

      import startCase from '@lodash/startCase';
      
      const el = document.createElement('h1');
      const words = "hello, world";
      const text = document.createTextNode(words);
      el.appendChild(text);
      
      document.body.appendChild(el);
      

      Finally, call startCase with the words variable as an argument inside of document.createTextNode:

      hello.js

      import startCase from '@lodash/startCase';
      
      const el = document.createElement('h1');
      const words = "hello, world";
      const text = document.createTextNode(startCase(words));
      el.appendChild(text);
      
      document.body.appendChild(el);
      

      If you closed your webserver, open a new terminal window or tab and run npx serve to restart the server. Then navigate to http://localhost:5000 in a web browser to view the changes.

      When you preview the code in a browser, open the developer console. When you do, you’ll see an error:

      Output

      Uncaught SyntaxError: Cannot use import statement outside a module

      Module Error

      Since the code is using import statements, you’ll need to modify the <script> tag inside of index.html to handle JavaScript that is now split between multiple files. One file is the original code you wrote. The other file is the code imported from lodash. JavaScript code that imports other code are called modules.

      Close hello.js and open index.html:

      To run the script as a module, change the value of the type attribute on the <script> tag to module:

      hello.js

      <!DOCTYPE html>
      <html lang="en-US">
      <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <title>Hello World</title>
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <script type="module" src="https://www.digitalocean.com/community/tutorials/./hello.js"></script>
      </head>
      <body>
      </body>
      </html>
      

      Notice, you also removed the defer attribute. JavaScript modules will not execute until the page is parsed.

      Open your browser and you’ll still see an error`:

      Output

      Uncaught TypeError: Failed to resolve module specifier "@lodash/startCase". Relative references must start with either "/", "./", or "../"

      Unknown specifier

      Now the problem is not the import statement. The problem is the browser doesn’t know what the import statement means. It’s expecting to find code on the webserver, so it looks for a file relative to the current file. To solve that problem, you’ll need a new tool called import maps.

      In this step, you learned how to modify your JavaScript code to load external libraries using ES imports. You also modified the HTML script tag to handle JavaScript modules.

      In the next step, you’ll tell the browser how to find code using import maps.

      Step 3 — Loading External Code with Import Maps

      In this step, you’ll learn how to create import maps to tell your browser where to find external code. You’ll also learn how to import module code and see how code is lazy-loaded in a browser.

      An import map is a JavaScript object where the key is the name of the import (@lodash/startCase) and the value is the location of the code.

      Inside of index.html add a new script tag with a type of importmap. Inside of the script tag, create a JavaScript object with a key of imports. The value will be another object:

      hello.js

      <!DOCTYPE html>
      <html lang="en-US">
      <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <title>Hello World</title>
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <script type="importmap">
          {
            "imports": {
            }
          }
        </script>
        <script type="module" src="https://www.digitalocean.com/community/tutorials/./hello.js"></script>
      </head>
      <body>
      </body>
      </html>
      

      Be sure that you do not add any trailing commas in the object. The browser will not know how to handle them.

      Now that you have your basic object, you can add the location of your code. The structure of an import map will be like this:

      {
          "imports": {
              "nameOfImport": "locationOfCode",
              "nameOfSecondImport": "secondLocation"
          }
      }
      

      You already know the name of your import @lodash/startCase, but now you need to find where a location to point the import map to.

      A great resource is unpkg. Unpkg is a content delivery network (CDN) for any package in npm. If you can npm install a package, you should be able to load it via unpkg. Unpkg also includes a browsing option that can help you find the specific file you need.

      To find the startCase code, open https://unpkg.com/browse/lodash-es@4.17.21/ in a browser. Notice the word browse in the URL. This gives you a graphical way to look through the directory, but you should not add the path to your import map since it serves up an HTML page and not the raw JavaScript file.

      Also, note that you are browsing lodash-es and not lodash. This is the lodash library exported as ES modules, which is what you will need in this case.

      Browse the code and you’ll notice there is a file called startCase.js. This code imports other functions and uses them to convert the first letter of each word to upper case:

      import createCompounder from './_createCompounder.js';
      import upperFirst from './upperFirst.js';
      
      /**
       * Converts `string` to
       * [start case](https://en.wikipedia.org/wiki/Letter_case#Stylistic_or_specialised_usage).
       *
       * @static
       * @memberOf _
       * @since 3.1.0
       * @category String
       * @param {string} [string=''] The string to convert.
       * @returns {string} Returns the start cased string.
       * @example
       *
       * _.startCase('--foo-bar--');
       * // => 'Foo Bar'
       *
       * _.startCase('fooBar');
       * // => 'Foo Bar'
       *
       * _.startCase('__FOO_BAR__');
       * // => 'FOO BAR'
       */
      var startCase = createCompounder(function(result, word, index) {
        return result + (index ? ' ' : '') + upperFirst(word);
      });
      
      export default startCase;
      

      The browser will follow the import statements and import every file necessary.

      Now that you have a location for your import map, update the file import map with the new URL. Inside of index.html, add @lodash/startCase along with the URL. Once again, be sure to remove browse:

      index.html

      <!DOCTYPE html>
      <html lang="en-US">
      <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <title>Hello World</title>
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <script type="importmap">
          {
            "imports": {
              "@lodash/startCase": "https://unpkg.com/lodash-es@4.17.21/startCase.js"
            }
          }
        </script>
        <script type="module" src="https://www.digitalocean.com/community/tutorials/./hello.js"></script>
      </head>
      <body>
      </body>
      </html>
      

      Save the file. Refresh your browser and you will see Hello World.

      Working

      NOTE: Import maps are not yet widely supported. Open the code in the latest version of Edge or Chrome or check out the latest supported browsers.

      Your browser shows “Hello World”, but now something much more interesting is happening. Open the browser console in your browser, select Inspect Elements, and switch to the Network tab.

      After opening the Network tab, refresh the page and you’ll see that the browser is loading the code dynamically. Whenever it finds a new import statement, it imports the relevant code:

      Network Tab

      More importantly, notice that all of the code is loaded lazily. That means that the browser does not import any code until it is specifically needed. For example, even though startCase is in the import map and the import map is defined before the script for hello.js, it is not loaded until after hello.js loads and imports the code.

      If you were to add other entries in your import map, the browser would not load them at all since they are never imported into code. The import map is a map of locations, and doesn’t import any code itself.

      One major problem is that import maps are not yet fully supported by all browsers. And even when they are supported, some users may not use a supported browser. Fortunately, there are different projects that use the import map syntax while adding full browser support.

      In this step you created an import map. You also learned how to import module code and how the code will be lazy loaded in the browser. In the next step, you’ll import code using SystemJS.

      Step 4 — Building Cross-Browser Support with SystemJS

      In this step, you’ll use import maps across all browsers using SystemJS. You’ll export code as a SystemJS build and how to set the import map type to use SystemJS format. By the end of this step, you’ll be able to take advantage of import maps in any browser and will have a foundation for building more complex applications and microfrontends.

      Import maps will remove many of the complicated build steps from an application, but they are not yet widely supported. Further, not all libraries are built as ES modules so there is some limitation to how you can use them in your code.

      Fortunately, there is a project called SystemJS that can use create import maps for cross-browser support and use a variety of package builds.

      The lodash library is convenient because it is compiled in an ES format, but that’s not the case for most libraries. Many libraries are exported in other formats. One of the most common is the Universal Module Definition or UMD. This format works in both browsers and node modules.

      A major difference is that unlike the ES imports, a UMD build typically combines all of the code into a single file. The file will be a lot larger and you’ll end up with more code then you’ll probably execute.

      To update your project to use SystemJS and a UMD build of lodash, first open hello.js:

      Change the import statement to import the startCase function directly from lodash.

      hello.js

      import { startCase } from 'lodash';
      
      const el = document.createElement('h1');
      const words = "hello, world";
      const text = document.createTextNode(startCase(words));
      el.appendChild(text);
      
      document.body.appendChild(el);
      

      Save and close the file.

      Next, to build the file as a SystemJS build, you will need a simple build step. You can use another build tool such as webpack, but in this example you’ll use rollup.

      First, initialize the project to create a package.json file. Add the -y flag to accept all of the defaults:

      npm init -y
      

      After the command runs you’ll see a success output:

      {
        "name": "hello",
        "version": "1.0.0",
        "description": "",
        "main": "index.js",
        "devDependencies": {},
        "scripts": {
          "test": "echo "Error: no test specified" && exit 1"
        },
        "keywords": [],
        "author": "",
        "license": "ISC",
        "homepage": ""
      }
      

      Note: Your output may be slightly different depending on the version of npm you are using.

      Next, install rollup as a devDepenceny:

      npm install --save-dev rollup
      

      After a moment, you will see a success message:

      + rollup@2.56.2
      added 1 package from 1 contributor and audited 2 packages in 6.85s
      found 0 vulnerabilities
      

      Next, create a simple build configuration. Open a new file called rollup.config.js:

      Then add a configuration that will output the code in SystemJS format:

      rollup.config.js

      export default {
        external: ["lodash"],
        input: ["hello.js"],
        output: [
          {
            dir: "public",
            format: "system",
            sourcemap: true
          }
        ]
      };
      

      The external key tells rollup not to include any of the lodash code in the final build. SystemJS will load that code dynamically when it is imported.

      The input is the location of the root file. The output tells rollup where to put the final code and the format it should use which in this case is system.

      Save and close the file.

      Now that you have a build step, you’ll need to add a task to run it. Open package.json:

      In the scripts object, add a script called build that will run rollup -c. Change the main key to hello.js:

      package.json

      {
        "name": "hello",
        "version": "1.0.0",
        "description": "",
        "main": "hello.js",
        "devDependencies": {
          "rollup": "^2.56.2"
        },
        "scripts": {
          "build": "rollup -c"
        },
        "keywords": [],
        "author": "",
        "license": "ISC",
        "homepage": ""
      }
      

      Save and close the file, and run the build command:

      npm run build
      

      The command will run briefly, then you will see a success message:

      > rollup -c
      
      
      hello.js → public...
      created public in 21ms
      
      

      You will also see a new directory called public that will contain the built file. If you open public/hello.js you’ll see your project compiled in a system format.

      The file will look like this. It’s similar to hello.js with a surrounding System.register method. In addtion, lodash is in an array. This will tell SystemJS to load the external library during run time. One of the maintainers created a video that further explains the module format.

      public/hello.js

      System.register(['lodash'], function () {
          'use strict';
          var startCase;
          return {
              setters: [function (module) {
                  startCase = module.startCase;
              }],
              execute: function () {
      
                  const el = document.createElement('h1');
                  const words = "hello, world";
                  const text = document.createTextNode(startCase(words));
                  el.appendChild(text);
      
                  document.body.appendChild(el);
      
              }
          };
      });
      //# sourceMappingURL=hello.js.map
      

      Save and close the file.

      The final step is to update your index.html to handle the new file:

      Open index.html

      First, you’ll need to import the SystemJS code. Use a regular <script> tag with the src attribute pointing to a CDN distribution.

      Put the <script> tag right below the import map:

      index.html

      <!DOCTYPE html>
      <html lang="en-US">
      <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <title>Hello World</title>
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <script type="importmap">
          {
            "imports": {
              "@lodash/startCase": "https://unpkg.com/lodash-es@4.17.21/startCase.js
            }
          }
        </script>
        <script src="https://cdn.jsdelivr.net/npm/systemjs/dist/system.js"></script>
        <script type="module" src="https://www.digitalocean.com/community/tutorials/./hello.js"></script>
      </head>
      <body>
      </body>
      </html>
      

      Next, you’ll need to update your import map. The format is similar to what you completed in Step 3, but there are three changes:

      • Update the import map type.
      • Update the reference to lodash.
      • Add a reference to the hello.js script.

      First, update the type. Since this is not the native browser version of an import map, but a systemjs version, change the type to systemjs-importmap:

      index.html

      <!DOCTYPE html>
      <html lang="en-US">
      <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <title>Hello World</title>
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <script type="systemjs-importmap">
          {
            "imports": {
              "@lodash/startCase": "https://unpkg.com/lodash-es@4.17.21/startCase.js
            }
          }
        </script>
        <script src="https://cdn.jsdelivr.net/npm/systemjs/dist/system.js"></script>
        <script type="module" src="https://www.digitalocean.com/community/tutorials/./hello.js"></script>
      </head>
      <body>
      </body>
      </html>
      

      Next, update the references. Change @lodash/startCase to lodash. You’ll be importing the full library. Then change the location to the UMD build at unpkg.

      Then add a new entry for hello and point that to the compiled version in the public directory:

      index.html

      ...
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <script type="systemjs-importmap">
          {
            "imports": {
              "hello": "./public/hello.js",
              "lodash": "https://unpkg.com/lodash@4.17.21/lodash.js"
            }
          }
        </script>
      ...
      

      Now that you are importing systemJS and have updated the import maps, all that’s left is to load the module.

      Change the type attribute on the script tag for the module to systemjs-module. Then change the src to import:hello. This will tell systemjs to load the hello script and execute:

      hello.js

      ...
        <script type="systemjs-importmap">
          {
            "imports": {
              "hello": "./public/hello.js",
              "lodash": "https://unpkg.com/lodash@4.17.21/lodash.js"
            }
          }
        </script>
        <script src="https://cdn.jsdelivr.net/npm/systemjs/dist/system.js"></script>
        <script type="systemjs-module" src="https://www.digitalocean.com/community/tutorials/import:hello"></script>
      </head>
      ...
      

      Save and close the file.

      When you do, the browser will refresh and you’ll see Hello World.

      Unlike native import maps, this will work in any browser. Here’s the result in FireFox:

      Hello in firefox

      If you look at the Network tab. You’ll see that as with import maps, the code is lazy loaded as needed:

      Firefox network tab

      In this step, you used import maps across browsers with SystemJS. You changed your script to use the UMD build of lodash, created a rollup build to output the code in system format, and changed the import map and module types to work with SystemJS

      Conclusion

      In this tutorial you used import maps to dynamically load JavaScript code. You rendered an application that dynamically loaded an external library without any build step. Then, you created a build process to generate your code in SystemJS format so that you can use import maps across all browsers.

      Import maps give you opportunities to start breaking large projects into smaller independent pieces called microfronentends. You also do not need to limit yourself to statically defined maps as you learned in this tutorial; you can also create dynamic import maps that can load from other scripts. You also can use a single import map for multiple projects by using scopes to define different versions of a dependency for different scripts that import them.

      There are new features in progress and you can follow them on the official spec.



      Source link