One place for hosting & domains

      Nodejs

      How To Process Images in Node.js With Sharp


      The author selected the Diversity in Tech Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Digital image processing is a method of using a computer to analyze and manipulate images. The process involves reading an image, applying methods to alter or enhance the image, and then saving the processed image. It’s common for applications that handle user-uploaded content to process images. For example, if you’re writing a web application that allows users to upload images, users may upload unnecessary large images. This can negatively impact the application load speed, and also waste your server space. With image processing, your application can resize and compress all the user-uploaded images, which can significantly improve your application performance and save your server disk space.

      Node.js has an ecosystem of libraries you can use to process images, such as sharp, jimp, and gm module. This article will focus on the sharp module. sharp is a popular Node.js image processing library that supports various image file formats, such as JPEG, PNG, GIF, WebP, AVIF, SVG and TIFF.

      In this tutorial, you’ll use sharp to read an image and extract its metadata, resize, change an image format, and compress an image. You will then crop, grayscale, rotate, and blur an image. Finally, you will composite images, and add text on an image. By the end of this tutorial, you’ll have a good understanding of how to process images in Node.js.

      Prerequisites

      To complete this tutorial, you’ll need:

      Step 1 — Setting Up the Project Directory and Downloading Images

      Before you start writing your code, you need to create the directory that will contain the code and the images you’ll use in this article.

      Open your terminal and create the directory for the project using the mkdir command:

      Move into the newly created directory using the cd command:

      Create a package.json file using npm init command to keep track of the project dependencies:

      The -y option tells npm to create the default package.json file.

      Next, install sharp as a dependency:

      You will use the following three images in this tutorial:

      Digitalocean maskot sammy
      Underwater ocean scene
      sammy with a transparent background

      Next, download the images in your project directory using the curl command.

      Use the following command to download the first image. This will download the image as sammy.png:

      • curl -O https://www.xpresservers.com/wp-content/webpc-passthru.php?src=https://www.xpresservers.com/wp-content/uploads/2021/09/How-To-Process-Images-in-Nodejs-With-Sharp.png&nocache=1

      Next, download the second image with the following command. This will download the image as underwater.png:

      • curl -O https://www.xpresservers.com/wp-content/webpc-passthru.php?src=https://www.xpresservers.com/wp-content/uploads/2021/09/1631157332_451_How-To-Process-Images-in-Nodejs-With-Sharp.png&nocache=1

      Finally, download the third image using the following command. This will download the image as sammy-transparent.png:

      • curl -O https://www.xpresservers.com/wp-content/webpc-passthru.php?src=https://www.xpresservers.com/wp-content/uploads/2021/09/1631157333_547_How-To-Process-Images-in-Nodejs-With-Sharp.png&nocache=1

      With the project directory and the dependencies set up, you’re now ready to start processing images.

      In this section, you’ll write code to read an image and extract its metadata. Image metadata is text embedded into an image, which includes information about the image such as its type, width, and height.

      To extract the metadata, you’ll first import the sharp module, create an instance of sharp, and pass the image path as an argument. After that, you’ll chain the metadata() method to the instance to extract the metadata and log it into the console.

      To do this, create and open readImage.js file in your preferred text editor. This tutorial uses a terminal text editor called nano:

      Next, require in sharp at the top of the file:

      process_images/readImage.js

      const sharp = require("sharp");
      

      sharp is a promise-based image processing module. When you create a sharp instance, it returns a promise. You can resolve the promise using the then method or use async/await, which has a cleaner syntax.

      To use async/await syntax, you’ll need to create an asynchronous function by placing the async keyword at the beginning of the function. This will allow you to use the await keyword inside the function to resolve the promise returned when you read an image.

      In your readImage.js file, define an asynchronous function, getMetadata(), to read the image, extract its metadata, and log it into the console:

      process_images/readImage.js

      const sharp = require("sharp");
      
      async function getMetadata() {
        const metadata = await sharp("sammy.png").metadata();
        console.log(metadata);
      }
      
      

      getMetadata() is an synchronous function given the async keyword you defined before the function label. This lets you use the await syntax within the function. The getMetadata() function will read an image and return an object with its metadata.

      Within the function body, you read the image by calling sharp() which takes the image path as an argument, here with sammy.png.

      Apart from taking an image path, sharp() can also read image data stored in a Buffer, Uint8Array, or Uint8ClampedArray provided the image is JPEG, PNG, GIF, WebP, AVIF, SVG or TIFF.

      Now, when you use sharp() to read the image, it creates a sharp instance. You then chain the metadata() method of the sharp module to the instance. The method returns an object containing the image metadata, which you store in the metadata variable and log its contents using console.log().

      Your program can now read an image and return its metadata. However, if the program throws an error during execution, it will crash. To get around this, you need to capture the errors when they occur.

      To do that, wrap the code within the getMetadata() function inside a try...catch block:

      process_images/readImage.js

      const sharp = require("sharp");
      
      async function getMetadata() {
        try {
          const metadata = await sharp("sammy.png").metadata();
          console.log(metadata);
        } catch (error) {
          console.log(`An error occurred during processing: ${error}`);
        }
      }
      

      Inside the try block, you read an image, extract and log its metadata. When an error occurs during this process, execution skips to the catch section and logs the error preventing the program from crashing.

      Finally, call the getMetadata() function by adding the highlighted line:

      process_images/readImage.js

      
      const sharp = require("sharp");
      
      async function getMetadata() {
        try {
          const metadata = await sharp("sammy.png").metadata();
          console.log(metadata);
        } catch (error) {
          console.log(`An error occurred during processing: ${error}`);
        }
      }
      
      getMetadata();
      

      Now, save and exit the file. Enter y to save the changes you made in the file, and confirm the file name by pressing ENTER or RETURN key.

      Run the file using the node command:

      You should see an output similar to this:

      Output

      { format: 'png', width: 750, height: 483, space: 'srgb', channels: 3, depth: 'uchar', density: 72, isProgressive: false, hasProfile: false, hasAlpha: false }

      Now that you’ve read an image and extracted its metadata, you’ll now resize an image, change its format, and compress it.

      Step 3 — Resizing, Changing Image Format, and Compressing Images

      Resizing is the process of altering an image dimension without cutting anything from it, which affects the image file size. In this section, you’ll resize an image, change its image type, and compress the image. Image compression is the process of reducing an image file size without losing quality.

      First, you’ll chain the resize() method from the sharp instance to resize the image, and save it in the project directory. Second, you’ll chain the format() method to the resized image to change its format from png to jpeg. Additionally, you will pass an option to the format() method to compress the image and save it to the directory.

      Create and open resizeImage.js file in your text editor:

      Add the following code to resize the image to 150px width and 97px height:

      process_images/resizeImage.js

      const sharp = require("sharp");
      
      async function resizeImage() {
        try {
          await sharp("sammy.png")
            .resize({
              width: 150,
              height: 97
            })
            .toFile("sammy-resized.png");
        } catch (error) {
          console.log(error);
        }
      }
      
      resizeImage();
      

      The resizeImage() function chains the sharp module’s resize() method to the sharp instance. The method takes an object as an argument. In the object, you set the image dimensions you want using the width and height property. Setting the width to 150 and the height to 97 will make the image 150px wide, and 97px tall.

      After resizing the image, you chain the sharp module’s toFile() method, which takes the image path as an argument. Passing sammy-resized.png as an argument will save the image file with that name in the working directory of your program.

      Now, save and exit the file. Run your program in the terminal:

      You will get no output, but you should see a new image file created with the name sammy-resized.png in the project directory.

      Open the image on your local machine. You should see an image of Sammy 150px wide and 97px tall:

      image resized to 150px width and 97px height

      Now that you can resize an image, next you’ll convert the resized image format from png to jpeg, compress the image, and save it in the working directory. To do that, you will use toFormat() method, which you’ll chain after the resize() method.

      Add the highlighted code to change the image format to jpeg and compress it:

      process_images/resizeImage.js

      const sharp = require("sharp");
      
      async function resizeImage() {
        try {
          await sharp("sammy.png")
            .resize({
              width: 150,
              height: 97
            })
            .toFormat("jpeg", { mozjpeg: true })
            .toFile("sammy-resized-compressed.jpeg");
        } catch (error) {
          console.log(error);
        }
      }
      
      resizeImage();
      

      Within the resizeImage() function, you use the toFormat() method of the sharp module to change the image format and compress it. The first argument of the toFormat() method is a string containing the image format you want to convert your image to. The second argument is an optional object containing output options that enhance and compress the image.

      To compress the image, you pass it a mozjpeg property that holds a boolean value. When you set it to true, sharp uses mozjpeg defaults to compress the image without sacrificing quality. The object can also take more options; see the sharp documentation for more details.

      Note: Regarding the toFormat() method’s second argument, each image format takes an object with different properties. For example, mozjpeg property is accepted only on JPEG images.

      However, other image formats have equivalents options such quality, compression, and lossless. Make sure to refer to the documentation to know what kind of options are acceptable for the image format you are compressing.

      Next, you pass the toFile() method a different filename to save the compressed image as sammy-resized-compressed.jpeg.

      Now, save and exit the file, then run your code with the following command:

      You will receive no output, but an image file sammy-resized-compressed.jpeg is saved in your project directory.

      Open the image on your local machine and you will see the following image:

      Sammy image resized and compressed

      With your image now compressed, check the file size to confirm your compression is successful. In your terminal, run the du command to check the file size for sammy.png:

      -h option produces human-readable output showing you the file size in kilobytes, megabytes and many more.

      After running the command, you should see an output similar to this:

      Output

      120K sammy.png

      The output shows that the original image is 120 kilobytes.

      Next, check the file size for sammy-resized.png:

      After running the command, you will see the following output:

      Output

      8.0K sammy-resized.png

      sammy-resized.png is 8 kilobytes down from 120 kilobytes. This shows that the resizing operation affects the file size.

      Now, check the file size for sammy-resized-compressed.jpeg:

      • du -h sammy-resized-compressed.jpeg

      After running the command, you will see the following output:

      Output

      4.0K sammy-resized-compressed.jpeg

      The sammy-resized-compressed.jpeg is now 4 kilobytes down from 8 kilobytes, saving you 4 kilobytes, showing that the compression worked.

      Now that you’ve resized an image, changed its format and compressed it, you will crop and grayscale the image.

      Step 4 — Cropping and Converting Images to Grayscale

      In this step, you will crop an image, and convert it to grayscale. Cropping is the process of removing unwanted areas from an image. You’ll use the extend() method to crop the sammy.png image. After that, you’ll chain the grayscale() method to the cropped image instance and convert it to grayscale.

      Create and open cropImage.js in your text editor:

      In your cropImage.js file, add the following code to crop the image:

      process_images/cropImage.js

      const sharp = require("sharp");
      
      async function cropImage() {
        try {
          await sharp("sammy.png")
            .extract({ width: 500, height: 330, left: 120, top: 70  })
            .toFile("sammy-cropped.png");
        } catch (error) {
          console.log(error);
        }
      }
      
      cropImage();
      

      The cropImage() function is an asynchronous function that reads an image and returns your image cropped. Within the try block, a sharp instance will read the image. Then, the sharp module’s extract() method chained to the instance takes an object with the following properties:

      • width: the width of the area you want to crop.
      • height: the height of the area you want to crop.
      • top: the vertical position of the area you want to crop.
      • left: the horizontal position of the area you want to crop.

      When you set the width to 500 and the height to 330, imagine that sharp creates a transparent box on top of the image you want to crop. Any part of the image that fits in the box will remain, and the rest will be cut:

      image showing the cropping area

      The top and left properties control the position of the box. When you set left to 120, the box is positioned 120px from the left edge of the image, and setting top to 70 positions the box 70px from the top edge of the image.

      The area of the image that fits within the box will be extracted out and saved into sammy-cropped.png as a separate image.

      Save and exit the file. Run the program in the terminal:

      The output won’t be shown but the image sammy-cropped.png will be saved in your project directory.

      Open the image on your local machine. You should see the image cropped:

      image cropped

      Now that you cropped an image, you will convert the image to grayscale. To do that, you’ll chain the grayscale method to the sharp instance. Add the highlighted code to convert the image to grayscale:

      process_images/cropImage.js

      const sharp = require("sharp");
      
      async function cropImage() {
        try {
          await sharp("sammy.png")
            .extract({ width: 500, height: 330, left: 120, top: 70 })
            .grayscale()
            .toFile("sammy-cropped-grayscale.png");
        } catch (error) {
          console.log(error);
        }
      }
      
      cropImage();
      

      The cropImage() function converts the cropped image to grayscale by chaining the sharp module’s grayscale() method to the sharp instance. It then saves the image in the project directory as sammy-cropped-grayscale.png.

      Press CTRL+X to save and exit the file.

      Run your code in the terminal:

      Open sammy-cropped-grayscale.png on your local machine. You should now see the image in grayscale:

      image cropped and grayscaled

      Now that you’ve cropped and extracted the image, you’ll work with rotating and blurring it.

      Step 5 — Rotating and Blurring Images

      In this step, you’ll rotate the sammy.png image at a 33 degrees angle. You’ll also apply a gaussian blur on the rotated image. A gaussian blur is a technique of blurring an image using the Gaussian function, which reduces the noise level and detail on an image.

      Create a rotateImage.js file in your text editor:

      In your rotateImage.js file, write the following code block to create a function that rotates sammy.png to an angle of 33 degrees:

      process_images/rotateImage.js

      const sharp = require("sharp");
      
      async function rotateImage() {
        try {
          await sharp("sammy.png")
            .rotate(33, { background: { r: 0, g: 0, b: 0, alpha: 0 } })
            .toFile("sammy-rotated.png");
        } catch (error) {
          console.log(error);
        }
      }
      
      rotateImage();
      

      The rotateImage() function is an asynchronous function that reads an image and will return the image rotated to an angle of 33 degrees. Within the function, the rotate() method of the sharp module takes two arguments. The first argument is the rotation angle of 33 degrees. By default, sharp makes the background of the rotated image black. To remove the black background, you pass an object as a second argument to make the background transparent.

      The object has a background property which holds an object defining the RGBA color model. RGBA stands for red, green, blue, and alpha.

      • r: controls the intensity of the red color. It accepts an integer value of 0 to 255. 0 means the color is not being used, and 255 is red at its highest.

      • g: controls the intensity of the green color. It accepts an integer value of 0-255. 0 means that the color green is not used, and 255 is green at its highest.

      • b: controls the intensity of blue. It also accepts an integer value between 0 and 255. 0 means that the blue color isn’t used, and 255 is blue at its highest.

      • alpha: controls the opacity of the color defined by r, g, and b properties. 0 or 0.0 makes the color transparent and 1 or 1.1 makes the color opaque.

      For the alpha property to work, you must make sure you define and set the values for r, g, and b. Setting the r, g, and b values to 0 creates a black color. To create a transparent background, you must define a color first, then you can set alpha to 0 to make it transparent.

      Now, save and exit the file. Run your script in the terminal:

      Check for the existence of sammy-rotated.png in your project directory. Open it on your local machine.

      You should see the image rotated to an angle of 33 degrees:

      image rotated 33 degrees

      Next, you’ll blur the rotated image. You’ll achieve that by chaining the blur() method to the sharp instance.

      Enter the highlighted code below to blur the image:

      process_images/rotateImage.js

      const sharp = require("sharp");
      
      async function rotateImage() {
        try {
          await sharp("sammy.png")
            .rotate(33, { background: { r: 0, g: 0, b: 0, alpha: 0 } })
            .blur(4)
            .toFile("sammy-rotated-blurred.png");
        } catch (error) {
          console.log(error);
        }
      }
      
      rotateImage();
      

      The rotateImage() function now reads the image, rotate it, and applies a gaussian blur to the image. It applies a gaussian blur to the image using the sharp module’s blur() method. The method accepts a single argument containing a sigma value between 0.3 and 1000. Passing it 4 will apply a gaussian blur with a sigma value of 4. After the image is blurred, you define a path to save the blurred image.

      Your script will now blur the rotated image with a sigma value of 4. Save and exit the file, then run the script in your terminal:

      After running the script, open sammy-rotated-blurred.png file on your local machine. You should now see the rotated image blurred:

      rotated image blurred

      Now that you’ve rotated and blurred an image, you’ll composite an image over another.

      Step 6 — Compositing Images Using composite()

      Image Composition is a process of combining two or more separate pictures to create a single image. This is done to create effects that borrow the best elements from the different photos. Another common use case is to watermark an image with a logo.

      In this section, you’ll composite sammy-transparent.png over the underwater.png. This will create an illusion of sammy swimming deep in the ocean. To composite the images, you’ll chain the composite() method to the sharp instance.

      Create and open the file compositeImage.js in your text editor:

      Now, create a function to composite the two images by adding the following code in the compositeImages.js file:

      process_images/compositeImages.js

      const sharp = require("sharp");
      
      async function compositeImages() {
        try {
          await sharp("underwater.png")
            .composite([
              {
                input: "sammy-transparent.png",
                top: 50,
                left: 50,
              },
            ])
            .toFile("sammy-underwater.png");
        } catch (error) {
          console.log(error);
        }
      }
      
      compositeImages()
      

      The compositeImages() function reads the underwater.png image first. Next, you chain the composite() method of the sharp module, which takes an array as an argument. The array contains a single object that reads the sammy-transparent.png image. The object has the following properties:

      • input: takes the path of the image you want to composite over the processed image. It also accepts a Buffer, Uint8Array, or Uint8ClampedArray as input.
      • top: controls the vertical position of the image you want to composite over. Setting top to 50 offsets the sammy-transparent.png image 50px from the top edge of the underwater.png image.
      • left: controls the horizontal position of the image you want to composite over another. Setting left to 50 offsets the sammy-transparent.png 50px from the left edge of the underwater.png image.

      The composite() method requires an image of similar size or smaller to the processed image.

      To visualize what the composite() method is doing, think of it like its creating a stack of images. The sammy-transparent.png image is placed on top of underwater.png image:

      a graphic showing an image stack

      The top and left values positions the sammy-transparent.png image relative to the underwater.png image.

      Save your script and exit the file. Run your script to create an image composition:

      node compositeImages.js
      

      Open sammy-underwater.png in your local machine. You should now see the sammy-transparent.png composited over the underwater.png image:

      an image composition

      You’ve now composited images using the composite() method. In the next step, you’ll use the composite() method to add text to an image.

      Step 7 — Adding Text on an Image

      In this step, you’ll write text on an image. At the time of writing, sharp doesn’t have a native way of adding text to an image. To add text, first, you’ll write code to draw text using Scalable Vector Graphics(SVG). Once you’ve created the SVG image, you’ll write code to composite the image with the sammy.png image using the composite method.

      SVG is an XML-based markup language for creating vector graphics for the web. You can draw text, or shapes such as circles, triangles, and as well as draw complex shapes such as illustrations, logos, etc. The complex shapes are created with a graphic tool like Inkscape which generates the SVG code. The SVG shapes can be rendered and scaled to any size without losing quality.

      Create and open the addTextOnImage.js file in your text editor.

      In your addTextOnImage.js file, add the following code to create an SVG container:

      process_images/addTextOnImage.js

      const sharp = require("sharp");
      
      async function addTextOnImage() {
        try {
          const width = 750;
          const height = 483;
          const text = "Sammy the Shark";
      
          const svgImage = `
          <svg width="${width}" height="${height}">
          </svg>
          `;
        } catch (error) {
          console.log(error);
        }
      }
      
      addTextOnImage();
      

      The addTextOnImage() function defines four variables: width, height, text, and svgImage. width holds the integer 750, and height holds the integer 483. text holds the string Sammy the Shark. This is the text that you’ll draw using SVG.

      The svgImage variable holds the svg element. The svg element has two attributes: width and height that interpolates the width and height variables you defined earlier. The svg element creates a transparent container according to the given width and height.

      You gave the svg element a width of 750 and height of 483 so that the SVG image will have the same size as sammy.png. This will help in making the text look centered on the sammy.png image.

      Next, you’ll draw the text graphics. Add the highlighted code to draw Sammy the Shark on the SVG container:

      process_images/addTextOnImage.js

      async function addTextOnImage() {
          ...
          const svg = `
          <svg width="${width}" height="${height}">
          <text x="50%" y="50%" text-anchor="middle" class="title">${text}</text>
          </svg>
          `;
        ....
      }
      

      The SVG text element has four attributes: x, y, text-anchor, and class. x and y define the position for the text you are drawing on the SVG container. The x attribute positions the text horizontally, and the y attribute positions the text vertically.

      Setting x to 50% draws the text in the middle of the container on the x-axis, and setting y to 50% positions the text in the middle on y-axis of the SVG image.

      The text-anchor aligns text horizontally. Setting text-anchor to middle will align the text on the center at the x coordinate you specified.

      class defines a class name on the text element. You’ll use the class name to apply CSS styles to the text element.

      ${text} interpolates the string Sammy the Shark stored in the text variable. This is the text that will be drawn on the SVG image.

      Next, add the highlighted code to style the text using CSS:

      process_images/addTextOnImage.js

          const svg = `
          <svg width="${width}" height="${height}">
            <style>
            .title { fill: #001; font-size: 70px; font-weight: bold;}
            </style>
            <text x="50%" y="50%" text-anchor="middle" class="title">${text}</text>
          </svg>
          `;
      

      In this code, fill changes the text color to black, font-size changes the font size, and font-weight changes the font weight.

      At this point, you have written the code necessary to draw the text Sammy the Shark with SVG. Next, you’ll save the SVG image as a png with sharp so that you can see how SVG is drawing the text. Once that is done, you’ll composite the SVG image with sammy.png.

      Add the highlighted code to save the SVG image as a png with sharp:

      process_images/addTextOnImage.js

          ....
          const svgImage = `
          <svg width="${width}" height="${height}">
          ...
          </svg>
          `;
          const svgBuffer = Buffer.from(svgImage);
          const image = await sharp(svgBuffer).toFile("svg-image.png");
        } catch (error) {
          console.log(error);
        }
      }
      
      addTextOnImage();
      

      Buffer.from() creates a Buffer object from the SVG image. A buffer is a temporary space in memory that stores binary data.

      After creating the buffer object, you create a sharp instance with the buffer object as input. In addition to an image path, sharp also accepts a buffer, Uint9Array, or Uint8ClampedArray.

      Finally, you save the SVG image in the project directory as svg-image.png.

      Here is the complete code:

      process_images/addTextOnImage.js

      const sharp = require("sharp");
      
      async function addTextOnImage() {
        try {
          const width = 750;
          const height = 483;
          const text = "Sammy the Shark";
      
          const svgImage = `
          <svg width="${width}" height="${height}">
            <style>
            .title { fill: #001; font-size: 70px; font-weight: bold;}
            </style>
            <text x="50%" y="50%" text-anchor="middle" class="title">${text}</text>
          </svg>
          `;
          const svgBuffer = Buffer.from(svgImage);
          const image = await sharp(svgBuffer).toFile("svg-image.png");
        } catch (error) {
          console.log(error);
        }
      }
      
      addTextOnImage()
      

      Save and exit the file, then run your script with the following command:

      node addTextOnImage.js
      

      Note: If you installed Node.js using Option 2 — Installing Node.js with Apt Using a NodeSource PPA or Option 3 — Installing Node Using the Node Version Manager and getting the error fontconfig error: cannot load default config file: no such file: (null), install fontconfig to generate the font configuration file.

      Update your server’s package index, and after that, use apt install to install fontconfig.

      • sudo apt update
      • sudo apt install fontconfig

      Open svg-image.png on your local machine. You should now see the text Sammy the Shark rendered with a transparent background:

      svg text rendered

      Now that you’ve confirmed the SVG code draws the text, you will composite the text graphics onto sammy.png.

      Add the following highlighted code to composite the SVG text graphics image onto the sammy.png image.

      process_images/addTextOnImage.js

      const sharp = require("sharp");
      
      async function addTextOnImage() {
        try {
          const width = 750;
          const height = 483;
          const text = "Sammy the Shark";
      
          const svgImage = `
          <svg width="${width}" height="${height}">
            <style>
            .title { fill: #001; font-size: 70px; font-weight: bold;}
            </style>
            <text x="50%" y="50%" text-anchor="middle" class="title">${text}</text>
          </svg>
          `;
          const svgBuffer = Buffer.from(svgImage);
          const image = await sharp("sammy.png")
            .composite([
              {
                input: svgBuffer,
                top: 0,
                left: 0,
              },
            ])
            .toFile("sammy-text-overlay.png");
        } catch (error) {
          console.log(error);
        }
      }
      
      addTextOnImage();
      

      The composite() method reads the SVG image from the svgBuffer variable, and positions it 0 pixels from the top, and 0 pixels from the left edge of the sammy.png. Next, you save the composited image as sammy-text-overlay.png.

      Save and close your file, then run your program using the following command:

      Open sammy-text-overlay.png on your local machine. You should see text added over the image:

      text added on image

      You have now used the composite() method to add text created with SVG on another image.

      Conclusion

      In this article, you learned how to use sharp methods to process images in Node.js. First, you created an instance to read an image and used the metadata() method to extract the image metadata. You then used the resize() method to resize an image. Afterwards, you used the format() method to change the image type, and compress the image. Next, you proceeded to use various sharp methods to crop, grayscale, rotate, and blur an image. Finally, you used the composite() method to composite an image, and add text on an image.

      For more insight into additional sharp methods, visit the sharp documentation. If you want to continue learning Node.js, see How To Code in Node.js series.



      Source link

      How To Use console in Node.js


      Introduction

      In this article, we’ll learn how to use most methods available in the Node.js console class more effectively.

      Prerequisites

      This tutorial was verified with Chrome browser version 70.0.3538.77 and Node.js version 8.11.3.

      Using console.log, console.info, and console.debug

      The console.log method prints to standard out, whether this is the terminal or browser console.
      It outputs strings by default but can be used in conjunction with template strings to modify what it returns.

      console.log(string, substitution)
      

      console.info and console.debug methods are identical to console.log in their operation.

      You can use console.debug in the Firefox browser console by default but to use it in Chrome, you’ll have to set the log level to Verbose by toggling it on in the All levels menu.

      Screenshot depicting the developer tools with All levels menu selected and Verbose toggled on.

      The arguments in the template string are passed to util.format which then processes the arguments by replacing each substitution token with the respective converted value.

      The supported substitution tokens are:

      %s

      const msg = `Using the console class`;
      console.log('%s', msg);
      console.log(msg);
      

      This code would output the following:

      Output

      Using the console class Using the console class

      %s is the default substitution pattern.

      %d, %f, %i, %o

      const circle = (radius = 1) => {
        const profile = {};
        const pi = 22/7;
        profile.diameter = 2 * pi * radius;
        profile.circumference = pi * radius * 2;
        profile.area = pi * radius * 2;
        profile.volume = 4/3 * pi * radius^3;
      
        console.log('This circle has a radius of: %d cm', radius);
        console.log('This circle has a circumference of: %f cm', profile.diameter);
        console.log('This circle has an area of: %i cm^2', profile.area);
        console.log('The profile of this cirlce is: %o', profile);
        console.log('Diameter %d, Area: %f, Circumference %i', profile.diameter, profile.area, profile.circumference)
      }
      
      circle();
      

      This code would output the following:

      Output

      This circle has a radius of: 1 cm This circle has a circumference of: 6.285714285714286 cm This circle has an area of: 6 cm^2 The profile of this cirlce is: {diameter: 6.285714285714286, circumference: 6.285714285714286, area: 6.285714285714286, volume: 7} Diameter 6, Area: 6.285714285714286, Circumference 6
      • %d will be substituted by a digit (integer or float).
      • %f will be replaced by a float value.
      • %i will be replaced by an integer.
      • %o will be replaced by an Object.

      %o is especially handy because we don’t have to use JSON.stringify to expand our object because it shows all the object’s properties by default.

      Note that you can use as many token substitutions as you like. They’ll just be replaced in the same order as the arguments you pass.

      %c

      This substitution token applies CSS styles to the substituted text.

      console.log('LOG LEVEL: %c OK', 'color: green; font-weight: normal');
      console.log('LOG LEVEL: %c PRIORITY', 'color: blue; font-weight: medium');
      
      console.log('LOG LEVEL: %c WARN', 'color: red; font-weight: bold');
      console.log('ERROR HERE');
      

      This code would output the following:

      Screenshot of console output with OK in green, PRIORITY in blue, WARN in red.

      The text we pass to console.log after the %c substitution token is affected by the styles, but the text before is left as is without styling.

      Using console.table

      The first argument passed to it is the data to be returned in the form of a table. The second is an array of selected columns to be displayed.

      console.table(tabularData, [properties])
      

      This method will print the input passed to it formatted as a table then log the input object after the table representation.

      Arrays

      If an array is passed to it as data, each element in the array will be a row in the table.

      const books = ['The Silmarillion', 'The Hobbit', 'Unfinished Tales'];
      console.table(books);
      

      Screenshot of the books array displayed in a table format.

      With a simple array with a depth of 1, the first column in the table has the heading index. Under the index header in the first column are the array indexes and the items in the array are listed in the second column under the value header.

      This is what happens for a nested array:

      const authorsAndBooks = [['Tolkien', 'Lord of The Rings'],['Rutger', 'Utopia For Realists'], ['Sinek', 'Leaders Eat Last'], ['Eyal', 'Habit']];
      console.table(authorsAndBooks);
      

      Screenshot of the authorsAndBooks array displayed in a table format.

      Objects

      For objects with a depth of 1, the object keys will be listed under the index header and the values in the object under the second column header.

      const inventory = { apples: 200, mangoes: 50, avocados: 300, kiwis: 50 };
      console.table(inventory);
      

      Screenshot of the inventory object displayed in a table format.

      For nested objects:

      const forexConverter = { asia: { rupee: 1.39, renminbi: 14.59 , ringgit: 24.26 }, africa: { rand: 6.49, nakfa: 6.7 , kwanza:0.33 }, europe: { swissfranc: 101.60, gbp: 130, euro: 115.73 } };
      console.table(forexConverter);
      

      Screenshot of the forexConverter object displayed in a table format.

      Some more nested objects,

      const workoutLog = { Monday: { push: 'Incline Bench Press', pull: 'Deadlift'}, Wednesday: { push: 'Weighted Dips', pull: 'Barbell Rows'}};
      console.table(workoutLog);
      

      Screenshot of the workoutLog object displayed in a table format.

      Here, we specify that we only want to see data under the column push.

      console.table(workoutLog, 'push');
      

      Screenshot of the workoutLog object displayed in a table format but restricted to push exercises.

      To sort the data under a column, just click the column header.

      Pretty handy, eh?

      Try passing console.table an object with some values as arrays!

      Using console.dir

      The first argument passed to this function is the object to be logged while the second is an object containing options that will define how the resulting output is formatted or what properties in the object will be shown.

      What’s returned is an object formatted by node’s util.inspect function.

      Nested or child objects within the input object are expandable under disclosure triangles.

      console.dir(object, options);
      // where options = { showHidden: true ... }
      

      Let’s see this in action.

      const user = {
        details: {
          name: {
            firstName: 'Immanuel',
            lastName: 'Kant'
          },
          height: `1.83m"`,
          weight: '90kg',
          age: '80',
          occupation: 'Philosopher',
          nationality: 'German',
          books: [
            {
              name: 'Critique of Pure Reason',
              pub: '1781',
            },
            {
              name: 'Critique of Judgement',
              pub: '1790',
            },
            {
              name: 'Critique of Practical Reason',
              pub: '1788',
            },
            {
              name: 'Perpetual Peace',
              pub: '1795',
            },
          ],
          death: '1804'
        }
      }
      
      console.dir(user);
      

      Here it is in the Chrome console.

      Screenshot of the user object displayed in a hierarchical dir format.

      Using console.dirxml

      This function will render an interactive tree of the XML/HTML it is passed. It defaults to a Javascript object if it’s not possible to render the node tree.

      console.dirxml(object|nodeList);
      

      Much like console.dir, the rendered tree can be expanded through clicking disclosure triangles within which you can see child nodes.

      Its output is similar to that which we find under the Elements tab in the browser.

      This is how it looks when we pass in some HTML from a Wikipedia page.

      const toc = document.querySelector('#toc');
      console.dirxml(toc);
      

      Screenshot of a nested series of HTML elements from the table of contents of a Wikipedia page.

      Let’s pass in some HTML from a page on this website.

      console.dirxml(document)
      

      Screenshot of a nested series of HTML elements from the current website.

      This is how it looks when we pass in an object.

      Screenshot of the user object displayed in a hierarchical dirxml format.

      Try calling console.dir on some HTML and see what happens!

      Using console.assert

      The first argument passed to the function is a value to test as truthy. All other arguments passed are considered messages to be printed out if the value passed is not evaluated as truthy.

      The Node REPL will throw an error halting the execution of subsequent code.

      console.assert(value, [...messages])
      

      Here’s a basic example:

      • console.assert(false, 'Assertion failed');

      Output

      Assertion failed: Assertion failed

      Now, let’s have some fun. We’ll build a mini testing framework using console.assert

      const sum = (a = 0, b = 0) => Number(a) + Number(b);
      
      function test(functionName, actualFunctionResult, expected) {
        const actual = actualFunctionResult;
        const pass = actual === expected;
        console.assert(pass, `Assertion failed for ${functionName}`);
        return `Test passed ${actual} === ${expected}`;
      }
      
      console.log(test('sum', sum(1,1), 2)); // Test passed 2 === 2
      console.log(test('sum', sum(), 0));    // Test passed 0 === 0
      console.log(test('sum', sum, 2));      // Assertion failed for sum
      console.log(test('sum', sum(3,3), 4)); // Assertion failed for sum
      

      Run the above in your node REPL or browser console to see what happens.

      Using console.error and console.warn

      These two are essentially identical. They will both print whatever string is passed to them.

      However, console.warn prints out a triangle warn symbol before the message is passed:

      console.warn(string, substitution);
      

      Screenshot of console.warn(3).

      While console.error prints out a danger symbol before the message passed.

      console.error(string, substitution);
      

      Screenshot of console.error(2)

      Let’s note that string substitution can be applied in the same way as the console.log method.

      Here’s a mini logging function using console.error.

      const sum = (a = 0, b = 0) => Number(a) + Number(b);
      
      function otherTest(actualFunctionResult, expected) {
        if (actualFunctionResult !== expected) {
          console.error(new Error(`Test failed ${actualFunctionResult} !== ${expected}`));
        } else {
          // pass
        }
      }
      
      otherTest(sum(1,1), 3);
      

      Screenshot of console.error Test failed 2 !== 3.

      Using console.trace(label)

      This console method will print the string Trace: followed by the label passed to the function then the stack trace to the current position of the function.

      function getCapital(country) {
        const capitalMap = {
          belarus: 'minsk', australia: 'canberra', egypt: 'cairo', georgia: 'tblisi', latvia: 'riga', samoa: 'apia'
        };
        console.trace('Start trace here');
        return Object.keys(capitalMap).find(item => item === country) ? capitalMap[country] : undefined;
      }
      
      console.log(getCapital('belarus'));
      console.log(getCapital('accra'));
      

      Screenshot of stack trace for belarus displaying minsk and accra displaying undefined.

      Using console.count(label)

      Count will begin and increment a counter of name label.

      Let’s build a word counter to see how it works.

      const getOccurences = (word = 'foolish') => {
        const phrase = `Oh me! Oh life! of the questions of these recurring, Of the endless trains of the faithless, of cities fill’d with the foolish, Of myself forever reproaching myself, for who more foolish than I, and who more faithless?`;
      
        let count = 0;
        const wordsFromPhraseArray = phrase.replace(/[,.!?]/igm, '').split(' ');
        wordsFromPhraseArray.forEach((element, idx) => {
          if (element === word) {
            count ++;
            console.count(word);
          }
        });
        return count;
      }
      
      getOccurences();
      
      getOccurences('foolish');
      

      Here, we see that the word foolish was logged twice. Once for each appearance of the word in the phrase.

      [secondary_label]
      foolish: 1
      foolish: 2
      2
      

      We could use this as a handy method to see how many times a function was called or how many times a line in our code was executed.

      Using console.countReset(label)

      As the name suggests, this resets a counter having a label set by the console.count method.

      const getOccurences = (word = 'foolish') => {
        const phrase = `Oh me! Oh life! of the questions of these recurring, Of the endless trains of the faithless, of cities fill’d with the foolish, Of myself forever reproaching myself, for who more foolish than I, and who more faithless?`;
      
        let count = 0;
        const wordsFromPhraseArray = phrase.replace(/[,.!?]/igm, '').split(' ');
        wordsFromPhraseArray.forEach((element, idx) => {
          if (element === word) {
            count ++;
            console.count(word);
            console.countReset(word);
          }
        });
        return count;
      }
      
      getOccurences();
      
      getOccurences('foolish');
      
      [secondary_label]
      foolish: 1
      foolish: 1
      2
      

      We can see that our getOccurences function returns 2 because there are indeed two occurences of the word foolish in the phrase but since our counter is reset at every match, it logs foolish: 1 twice.

      Using console.time(label) and console.timeEnd(label)

      The console.time function starts a timer with the label supplied as an argument to the function, while the console.timeEnd function stops a timer with the label supplied as an argument to the function.

      console.time('<timer-label>');
      console.timeEnd('<timer-label>');
      

      We can use it to figure out how much time it took to run an operation by passing in the same label name to both functions.

      const users = ['Vivaldi', 'Beethoven', 'Ludovico'];
      
      const loop = (array) => {
        array.forEach((element, idx) => {
          console.log(element);
        })
      }
      
      const timer = () => {
        console.time('timerLabel');
        loop(users);
        console.timeEnd('timerLabel');
      }
      
      timer();
      

      We can see the timer label displayed against time value after the timer is stopped.

      Output

      Vivaldi Beethoven Ludovico timerLabel: 0.69091796875ms

      It took the loop function 0.6909ms to finish looping through the array.

      Conclusion

      At last, we’ve come to the end of this tutorial.

      Note that this tutorial did not cover non-standard uses of the console class like console.profile, console.profileEnd, and console.timeLog.



      Source link

      How To Build a Rate Limiter With Node.js on App Platform


      The author selected the COVID-19 Relief Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Rate limiting manages your network’s traffic and limits the number of times someone repeats an operation in a given duration, such as using an API. A service without a layer of security against rate limit abuse is prone to overload and hampers your application’s proper operation for legitimate customers.

      In this tutorial, you will build a Node.js server that will check the IP address of the request and also calculate the rate of these requests by comparing the timestamp of requests per user. If an IP address crosses the limit you have set for the application, you will call Cloudflare’s API and add the IP address to a list. You will then configure a Cloudflare Firewall Rule that will ban all requests with IP addresses in the list.

      By the end of this tutorial, you will have built a Node.js project deployed on DigitalOcean’s App Platform that protects a Cloudflare routed domain with rate limiting.

      Prerequisites

      Before you begin this guide, you will need:

      Step 1 — Setting Up the Node.js Project and Deploying to DigitalOcean’s App Platform

      In this step, you will expand on your basic Express server, push your code to a GitHub repository, and deploy your application to App Platform.

      Open the project directory of the basic Express server with your code editor. Create a new file by the name .gitignore in the root directory of the project. Add the following lines to the newly created .gitignore file:

      .gitignore

      node_modules/
      .env
      

      The first line in your .gitignore file is a directive to git not to track the node_modules directory. This will enable you to keep your repository size small. The node_modules can be generated when required by running the command npm install. The second line prevents the environment variable file from being tracked. You will create the .env file in further steps.

      Navigate to your server.js in your code editor and modify the following lines of code:

      server.js

      ...
      app.listen(process.env.PORT || 3000, () => {
          console.log(`Example app is listening on port ${process.env.PORT || 3000}`);
      });
      

      The change to conditionally use PORT as an environment variable enables the application to dynamically have the server running on the assigned PORT or use 3000 as the fallback one.

      Note: The string in console.log() is wrapped within backticks(`) and not within quotes. This enables you to use template literals, which provides the capability to have expressions within strings.

      Visit your terminal window and run your application:

      Your browser window will display Successful response. In your terminal, you will see the following output:

      Output

      Example app is listening on port 3000

      With your Express server running successfully, you’ll now deploy to App Platform.

      First, initialize git in the root directory of the project and push the code to your GitHub account. Navigate to the App Platform dashboard in the browser and click on the Create App button. Choose the GitHub option and authorize with GitHub, if necessary. Select your project’s repository from the dropdown list of projects you want to deploy to App Platform. Review the configuration, then give a name to the application. For the purpose of this tutorial, select the Basic plan as you’ll work in the application’s development phase. Once ready, click Launch App.

      Next, navigate to the Settings tab and click on the section Domains. Add your domain routed via Cloudflare into the field Domain or Subdomain Name. Select the bullet You manage your domain to copy the CNAME record that you’ll use to add to your domain’s Cloudflare DNS account.

      With your application deployed to App Platform, head over to your domain’s dashboard on Cloudflare in a new tab as you will return to App Platform’s dashboard later. Navigate to the DNS tab. Click on the Add Record button and select CNAME as your Type, @ as the root, and paste in the CNAME you copied from the App Platform. Click on the Save button, then navigate to the Domains section under the Settings tab in your App Platform’s Dashboard and click on the Add Domain button.

      Click the Deployments tab to see the details of the deployment. Once deployment finishes, you can open your_domain to view it on the browser. Your browser window will display: Successful response. Navigate to the Runtime Logs tab on the App Platform dashboard, and you will get the following output:

      Output

      Example app is listening on port 8080

      Note: The port number 8080 is the default assigned port by the App Platform. You can override this by changing the configuration while reviewing the app before deployment.

      With your application now deployed to App Platform, let’s look at how to outline a cache to calculate requests to the rate limiter.

      Step 2 — Caching User’s IP Address and Calculating Requests Per Second

      In this step, you will store a user’s IP address in a cache with an array of timestamps to monitor the requests per second of each user’s IP address. A cache is temporary storage for data frequently used by an application. The data in a cache is usually kept in quick access hardware like RAM (Random-Access Memory). The fundamental goal of a cache is to improve data retrieval performance by decreasing the need to visit the slower storage layer underneath it. You will use three npm packages: node-cache, is-ip, and request-ip to aid in the process.

      The request-ip package captures the user’s IP address used to request the server. The node-cache package creates an in-memory cache which you will use to keep track of user’s requests. You’ll use the is-ip package used to check if an IP Address is IPv6 Address. Install the node-cache, is-ip, and request-ip package via npm on your terminal.

      • npm i node-cache is-ip request-ip

      Open the server.js file in your code editor and add following lines of code below const express = require('express');:

      server.js

      ...
      const requestIP = require('request-ip');
      const nodeCache = require('node-cache');
      const isIp = require('is-ip');
      ...
      

      The first line here grabs the requestIP module from request-ip package you installed. This module captures the user’s IP address used to request the server. The second line grabs the nodeCache module from the node-cache package. nodeCache creates an in-memory cache, which you will use to keep track of user’s requests per second. The third line takes the isIp module from the is-ip package. This checks if an IP address is IPv6 which you will format as per Cloudflare’s specification to use CIDR notation.

      Define a set of constant variables in your server.js file. You will use these constants throughout your application.

      server.js

      ...
      const TIME_FRAME_IN_S = 10;
      const TIME_FRAME_IN_MS = TIME_FRAME_IN_S * 1000;
      const MS_TO_S = 1 / 1000;
      const RPS_LIMIT = 2;
      ...
      

      TIME_FRAME_IN_S is a constant variable that will determine the period over which your application will average the user’s timestamps. Increasing the period will increase the cache size, hence consume more memory. The TIME_FRAME_IN_MS constant variable will also determine the period of time your application will average user’s timestamps, but in milliseconds. MS_TO_S is the conversion factor you will use to convert time in milliseconds to seconds. The RPS_LIMIT variable is the threshold limit of the application that will trigger the rate limiter, and change the value as per your application’s requirements. The value 2 in the RPS_LIMIT variable is a moderate value that will trigger during the development phase.

      With Express, you can write and use middleware functions, which have access to all HTTP requests coming to your server. To define a middleware function, you will call app.use() and pass it a function. Create a function named ipMiddleware as middleware.

      server.js

      ...
      const ipMiddleware = async function (req, res, next) {
          let clientIP = requestIP.getClientIp(req);
          if (isIp.v6(clientIP)) {
              clientIP = clientIP.split(':').splice(0, 4).join(':') + '::/64';
          }
          next();
      };
      app.use(ipMiddleware);
      
      ...
      

      The getClientIp() function provided by requestIP takes the request object, req from the middleware, as parameter. The .v6() function comes from the is-ip module and returns true if the argument passed to it is an IPv6 address. Cloudflare’s Lists requires the IPv6 address in /64 CIDR notation. You need to format the IPv6 address to follow the format: aaaa:bbbb:cccc:dddd::/64. The .split(':') method creates an array from the string containing the IP address splitting them by the character :. The .splice(0,4) method returns the first four elements of the array. The join(':') function returns a string from the array combined with the character :.

      The next() call directs the middleware to go to the next middleware function if there is one. In your example, it will take the request to the GET route /. This is important to include at the end of your function. Otherwise, the request will not move forward from the middleware.

      Initialize an instance of node-cache by adding the following variable below the constants:

      server.js

      ...
      const IPCache = new nodeCache({ stdTTL: TIME_FRAME_IN_S, deleteOnExpire: false, checkperiod: TIME_FRAME_IN_S });
      ...
      

      With the constant variable IPCache, you are overriding the default parameters native to nodeCache with the custom properties:

      • stdTTL: The interval in seconds after which a key-value pair of cache elements will be evicted from the cache. TTL stands for Time To Live, and is a measure of time after which cache expires.
      • deleteOnExpire: Set to false as you will write a custom callback function to handle the expired event.
      • checkperiod: The interval in seconds after which an automatic check for expired elements is triggered. The default value is 600, and as your application’s element expiry is set to a lesser value, the check for expiry will also happen sooner.

      For more information on the default parameters of node-cache, you will find the node-cache npm package’s docs page useful. The following diagram will help you to visualise how a cache stores data:

      Schematic Representation of Data Stored in Cache

      You will now create a new key-value pair for the new IP address and append to an existing key-value pair if an IP address exists in the cache. The value is an array of timestamps corresponding to each request made to your application. In your server.js file, create the updateCache() function below the IPCache constant variable to add the timestamp of the request to cache:

      server.js

      ...
      const updateCache = (ip) => {
          let IPArray = IPCache.get(ip) || [];
          IPArray.push(new Date());
          IPCache.set(ip, IPArray, (IPCache.getTtl(ip) - Date.now()) * MS_TO_S || TIME_FRAME_IN_S);
      };
      ...
      

      The first line in the function gets the array of timestamps for the given IP address, or if null, initializes with an empty array. In the following line, you are pushing the present timestamp caught by the new Date() function into the array. The .set() function provided by node-cache takes three arguments: key, value and the TTL. This TTL will override the standard TTL set by replacing the value of stdTTL from the IPCache variable. If the IP address already exists in the cache, you will use the existing TTL; else, you will set TTL as TIME_FRAME_IN_S.

      The TTL for the current key-value pair is calculated by subtracting the present timestamp from the expiry timestamp. The difference is then converted to seconds and passed as the third argument to the .set() function. The .getTtl() function takes a key and IP address as an argument and returns the TTL of the key-value pair as a timestamp. If the IP address does not exist in the cache, it will return undefined and use the fallback value of TIME_FRAME_IN_S.

      Note: You require the conversion timestamps from milliseconds to seconds as JavaScript stores them in milliseconds while the node-cache module uses seconds.

      In the ipMiddleware middleware, add the following lines after the if code block if (isIp.v6(clientIP)) to calculate the requests per second of the IP address calling your application:

      server.js

      ...
          updateCache(clientIP);
          const IPArray = IPCache.get(clientIP);
          if (IPArray.length > 1) {
              const rps = IPArray.length / ((IPArray[IPArray.length - 1] - IPArray[0]) * MS_TO_S);
              if (rps > RPS_LIMIT) {
                  console.log('You are hitting limit', clientIP);
              }
          }
      ...
      

      The first line adds the timestamp of the request made by the IP address to the cache by calling the updateCache() function you declared. The second line collects the array of timestamps for the IP address. If the number of elements in the array of timestamps is greater than one (calculating requests per second needs a minimum of two timestamps), and the requests per second are more than the threshold value you defined in the constants, you will console.log the IP address. The rps variable calculates the requests per second by dividing the number of requests with a time interval difference, and converts the units to seconds.

      Since you had defaulted the property deleteOnExpire to the value false in the IPCache variable, you will now need to handle the expired event manually. node-cache provides a callback function that triggers on expired event. Add the following lines of code below the IPCache constant variable:

      server.js

      ...
      IPCache.on('expired', (key, value) => {
          if (new Date() - value[value.length - 1] > TIME_FRAME_IN_MS) {
              IPCache.del(key);
          }
      });
      ...
      

      .on() is a callback function that accepts key and value of the expired element as the arguments. In your cache, value is an array of timestamps of requests. The highlighted line checks if the last element in the array is at least TIME_FRAME_IN_S in the past than the present time. As you are adding elements to your array of timestamps, if the last element in value is at least TIME_FRAME_IN_S in the past than the present time, the .del() function takes key as an argument and deletes the expired element from the cache.

      For the instances when some elements of the array are at least TIME_FRAME_IN_S in the past than the present time, you need to handle it by removing the expired items from the cache. Add the following code in the callback function after the if code block if (new Date() - value[value.length - 1] > TIME_FRAME_IN_MS).

      server.js

      ...
          else {
              const updatedValue = value.filter(function (element) {
                  return new Date() - element < TIME_FRAME_IN_MS;
              });
              IPCache.set(key, updatedValue, TIME_FRAME_IN_S - (new Date() - updatedValue[0]) * MS_TO_S);
          }
      ...
      

      The filter() array method native to JavaScript provides a callback function to filter the elements in your array of timestamps. In your case, the highlighted line checks for elements that are least TIME_FRAME_IN_S in the past than the present time. The filtered elements are then added to the updatedValue variable. This will update your cache with the filtered elements in the updatedValue variable and a new TTL. The TTL that matches the first element in the updatedValue variable will trigger the .on('expired') callback function when the cache removes the following element. The difference of TIME_FRAME_IN_S and the time expired since the first request’s timestamp in updatedValue calculates the new and updated TTL.

      With your middleware functions now defined, visit your terminal window and run your application:

      Then, visit localhost:3000 in your web browser. Your browser window will display: Successful response. Refresh the page repeatedly to hit the RPS_LIMIT. Your terminal window will display:

      Output

      Example app is listening on port 3000 You are hitting limit ::1

      Note: The IP address for localhost is shown as ::1. Your application will capture the public IP of a user when deployed outside localhost.

      Your application is now able to able to track the user’s requests and store the timestamps in the cache. In the next step, you will integrate Cloudflare’s API to set up the Firewall.

      Step 3 — Setting Up the Cloudflare Firewall

      In this step, you will set up Cloudflare’s Firewall to block IP Addresses when hitting the rate limit, create environment variables, and make calls to the Cloudflare API.

      Visit the Cloudflare dashboard in your browser, log in, and navigate to your account’s homepage. Open Lists under Configurations tab. Create a new List with your_list as the name.

      Note: The Lists section is available on your Cloudflare account’s dashboard page and not your Cloudflare domain’s dashboard page.

      Navigate to the Home tab and open your_domain’s dashboard. Open the Firewall tab and click on Create a Firewall rule under the Firewall Rules section. Give your_rule_name to the Firewall to identify it. In the Field, select IP Source Address from the dropdown, is in list for the Operator, and your_list for the Value. Under the dropdown for Choose an action, select Block and click Deploy.

      Create a .env file in the project’s root directory with the following lines to call Cloudflare API from your application:

      .env

      ACCOUNT_MAIL=your_cloudflare_login_mail
      API_KEY=your_api_key
      ACCOUNT_ID=your_account_id
      LIST_ID=your_list_id
      

      To get a value for API_KEY, navigate to the API Tokens tab on the My Profile section of your Cloudflare dashboard. Click View in the Global API Key section and enter your Cloudflare password to view it. Visit the Lists section under the Configurations tab on the account’s homepage. Click on Edit beside your_list list you created. Get the ACCOUNT_ID and LIST_ID from the URL of your_list in the browser. The URL is of the format below:
      https://dash.cloudflare.com/your_account_id/configurations/lists/your_list_id

      Warning: Make sure the content of .env is kept confidential and not made public. Make sure you have the .env file listed in the .gitignore file you created in Step 1.

      Install the axios and dotenv package via npm on your terminal.

      Open the server.js file in your code editor and the add following lines of code below the nodeCache constant variable:

      server.js

      ...
      const axios = require('axios');
      require('dotenv').config();
      ...
      

      The first line here grabs the axios module from axios package you installed. You will use this module to make network calls to Cloudflare’s API. The second line requires and configures the dotenv module to enable the process.env global variable that will define the values you placed in your .env file to server.js.

      Add the following to the if (rps > RPS_LIMIT) condition within ipMiddleware above console.log('You are hitting limit', clientIP) to call Cloudflare API.

      server.js

      ...
          const url = `https://api.cloudflare.com/client/v4/accounts/${process.env.ACCOUNT_ID}/rules/lists/${process.env.LIST_ID}/items`;
          const body = [{ ip: clientIP, comment: 'your_comment' }];
          const headers = {
              'X-Auth-Email': process.env.ACCOUNT_MAIL,
              'X-Auth-Key': process.env.API_KEY,
              'Content-Type': 'application/json',
          };
          try {
              await axios.post(url, body, { headers });
          } catch (error) {
              console.log(error);
          }
      ...
      

      You are now calling the Cloudflare API through the URL to add an item, in this case an IP address, to your_list. The Cloudflare API takes your ACCOUNT_MAIL and API_KEY in the header of the request with the key as X-Auth-Email and X-Auth-Key. The body of the request takes an array of objects with ip as the IP address to add to the list, and a comment with the value your_comment to identify the entry. You can modify value of comment with your own custom comment. The POST request made via axios.post() is wrapped in a try-catch block to handle errors if any, that may occur. The axios.post function takes the url, body and an object with headers to make the request.

      Change the clientIP variable within the ipMiddleware function when testing out the API requests with a test IP address like 198.51.100.0/24 as Cloudflare does not accept the localhost’s IP address in its Lists.

      server.js

      ...
      const clientIP = '198.51.100.0/24';
      ...
      

      Visit your terminal window and run your application:

      Then, visit localhost:3000 in your web browser. Your browser window will display: Successful response. Refresh the page repeatedly to hit the RPS_LIMIT. Your terminal window will display:

      Output

      Example app is listening on port 3000 You are hitting limit ::1

      When you have hit the limit, open the Cloudflare dashboard and navigate to the your_list’s page. You will see the IP address you put in the code added to your Cloudflare’s List named your_list. The Firewall page will display after pushing your changes to GitHub.

      Warning: Make sure to change the value in your clientIP constant variable to requestIP.getClientIp(req) before deploying or pushing the code to GitHub.

      Deploy your application by committing the changes and pushing the code to GitHub. As you have set up auto-deploy, the code from GitHub will automatically deploy to your DigitalOcean’s App Platform. As your .env file is not added to GitHub, you will need to add it to App Platform via the Settings tab at App-Level Environment Variables section. Add the key-value pair from your project’s .env file so your application can access its contents on the App Platform. After you save the environment variables, open your_domain in your browser after deployment finishes and refresh the page repeatedly to hit the RPS_LIMIT. Once you hit the limit, the browser will show Cloudflare’s Firewall page.

      Cloudflare's Error 1020 Page

      Navigate to the Runtime Logs tab on the App Platform dashboard, and you will view the following output:

      Output

      ... You are hitting limit your_public_ip

      You can open your_domain from a different device or via VPN to see that the Firewall bans only the IP address in your_list. You can delete the IP address from your_list through your Cloudflare dashboard.

      Note: Occasionally, it takes few seconds for the Firewall to trigger due to the cached response from the browser.

      You have set up Cloudflare’s Firewall to block IP Addresses when users are hitting the rate limit by making calls to the Cloudflare API.

      Conclusion

      In this article, you built a Node.js project deployed on DigitalOcean’s App Platform connected to your domain routed via Cloudflare. You protected your domain against rate limit misuse by configuring a Firewall Rule on Cloudflare. From here, you can modify the Firewall Rule to show JS Challenge or CAPTCHA instead of banning the user. The Cloudflare documentation details the process.



      Source link