One place for hosting & domains

      December 2021

      How to Use a Private Go Module in Your Own Project


      The author selected the Diversity in Tech Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      One beneficial aspect of Go’s ecosystem is that a large number of modules are open source. Since they’re open source they can be freely accessed, examined, used, and learned from. However, sometimes it’s necessary to make a private Go module for various reasons, such as keeping proprietary business logic internal to your company.

      In this tutorial, you will publish a private Go module, set up authentication to access a private module, and use a private Go module in a project.

      Prerequisites

      Distributing a Private Module

      Unlike many programming languages, Go distributes modules from repositories instead of a central package server. One benefit of this approach is that publishing a private module is very similar to publishing a public one. Instead of requiring a completely separate private package server, a Go private module is distributed via a private source code repository. Since most source code hosting options support this out of the box, there’s no need to set up an additional private server.

      In order to use a private module, you’ll need to have access to a private Go module. In this section, you’ll create and publish a private module you can use later in the tutorial to access a private module from another Go program.

      To create your new private Go module, start by cloning the private GitHub repository where it will live. As part of the prerequisites you created a private, empty repository named mysecret in your GitHub account and this is the one you will use for your private module. This repository can be cloned anywhere you’d like on your computer, but many developers tend to have a directory for their projects. In this tutorial, you’ll use a directory named projects.

      Make the projects directory and navigate to it:

      • mkdir projects
      • cd projects

      From the projects directory, run git clone to clone your private mysecret repository to your computer:

      Git will confirm it has cloned your module and may warn you that you have cloned an empty repository. If so, this is not something you need to worry about:

      Output

      Cloning into 'mysecret'... warning: You appear to have cloned an empty repository.

      Next, use cd to go into the new mysecret directory you cloned and use go mod init, along with the name of your private repository, to create a new Go module:

      • cd mysecret
      • go mod init github.com/your_github_username/mysecret

      Now that your module is created, it’s time to add a function you can use from another project. Use nano, or your favorite text editor, to open a file with the same name as your repository, such as mysecret.go. The name isn’t significant, and could be anything, but using the same name as the repository makes it easier to determine which file to look in first when working with a new module:

      In the mysecret.go file, name the package with the same name as your repository, then add a SecretProcess function to print the line Running the secret process! when called:

      projects/mysecret/mysecret.go

      package mysecret
      
      import "fmt"
      
      func SecretProcess() {
          fmt.Println("Running the secret process!")
      }
      

      Now that you have your private module created, you will publish it to your private repository for others to use. Since your private repository only allows you to access it initially, you’re able to control who has access to your private module. You might restrict access to yourself, but you could also give access to friends or coworkers as well.

      Since both private and public Go modules are source repositories, publishing a private Go module follows the same process as publishing a public one. To publish your new module, stage your changes in the current directory using the git add command, then commit those changes to your local repository with the git commit command:

      • git add .
      • git commit -m "Initial private module implementation"

      You will see a confirmation from Git that your initial commit has succeeded as well as a summary of the files included in the commit:

      Output

      [main (root-commit) bda059d] Initial private module implementation 2 files changed, 10 insertions(+) create mode 100644 go.mod create mode 100644 mysecret.go

      Now the only part left is to move your changes to your GitHub repository. Similar to a public module, use the git push command to publish your code:

      Git will then push your changes and make them available to anyone with access to your private repository:

      git push origin main
      Enumerating objects: 4, done.
      Counting objects: 100% (4/4), done.
      Delta compression using up to 8 threads
      Compressing objects: 100% (3/3), done.
      Writing objects: 100% (4/4), 404 bytes | 404.00 KiB/s, done.
      Total 4 (delta 0), reused 0 (delta 0), pack-reused 0
      To github.com:your_github_username/mysecret.git
       * [new branch]      main -> main
      

      As with a public Go module, you can also add versions to your private Go module. The Publishing a New Module Version section of the How to Distribute Go Modules tutorial includes information on how to do this.

      In this section, you created a new module with a SecretProcess function and published it to your private mysecret GitHub repository, making it a private Go module. In order to access this module from another Go program, though, you’ll need to configure Go so it knows how to access the module.

      Configuring Go to Access Private Modules

      While Go modules are commonly distributed from their source code repositories, the Go team also runs a few central Go module services to aid ensure modules continue to exist if something happens to the original repository. By default, Go is configured to use these services, but they can cause problems when you try to download a private module because they don’t have access to those modules. To tell Go that some import paths are private and that it shouldn’t try to use the central Go services, you can use the GOPRIVATE environment variable. The GOPRIVATE environment variable is a comma-separated list of import path prefixes where, when encountered, the Go tools will try to access them directly instead of going through the central services. One such example would be the private module you just created.

      In order to use the private module, you will tell Go which path to consider private by setting it in the GOPRIVATE variable. There are a few choices you can make when setting your GOPRIVATE variable values. One option is to set GOPRIVATE to github.com. This might not be what you’re looking for, though, because this would tell Go not to use the central services for any module hosted on github.com, including the ones that aren’t yours.

      The next option would be to set GOPRIVATE to only your own user path, such as github.com/your_github_username. This solves the problem of considering all of GitHub private, but at some point you may have public modules you’ve created that you’d like to download through the Go module mirror. Doing this wouldn’t cause any problems and would be a perfectly reasonable option, but you also have the option of getting even more specific.

      The most specific option would be setting GOPRIVATE to match the path of your module exactly, such as: github.com/your_github_username/mysecret. This solves both of the issues from the previous options, but also introduces the issue that you need to add each of your private repositories to GOPRIVATE individually, such as shown here:

      GOPRIVATE=github.com/your_github_username/mysecret,github.com/your_github_username/othersecret
      

      Choosing the best option for youself is a matter of weighing the pros and cons in your situation.

      Since you only have a single private module now, we’ll use the full repository name for the value. To set the GOPRIVATE=github.com/your_github_username/mysecret environment variable in your current terminal, use the export command:

      • export GOPRIVATE=github.com/your_github_username/mysecret

      If you’d like to double-check that it’s set, you can use the env command along with grep to check for the GOPRIVATE name:

      Output

      GOPRIVATE=github.com/your_github_username/mysecret

      Even though Go now knows your module is private, it’s still not quite enough to use the module yet. If you try to go get your private module into another module, you’ll likely see an error similar to:

      • go get github.com/your_github_username/mysecret

      Output

      go get: module github.com/your_github_username/mysecret: git ls-remote -q origin in /Users/your_github_username/go/pkg/mod/cache/vcs/2f8c...b9ea: exit status 128: fatal: could not read Username for 'https://github.com': terminal prompts disabled Confirm the import path was entered correctly. If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.

      This error message says Go tried to download your module, but it encountered something it still doesn’t have access to. Since Git is being used to download the module, it would usually ask you to enter your credentials. However, in this case, Go is calling Git for you and can’t prompt for them. At this point, to access your module you’ll need to provide a way for Git to retrieve your credentials without your immediate input.

      Providing Private Module Credentials for HTTPS

      One way to tell Git how to log in on your behalf is the .netrc file. Located in a user’s home directory, the .netrc file contains various host names as well as log in credentials for those hosts. It’s widely used by a number of tools, including Git.

      By default, when go get tries to download a module, it will try to use HTTPS first. However, as shown in the previous example, it’s not able to prompt you for your username and password. To give Git your credentials, you’ll need to have a .netrc that includes github.com in your home directory.

      To create a .netrc file on Linux, MacOS, or Windows Subsystem for Linux (WSL), open the .netrc file in your home directory (~/) so you can edit it:

      Next, create a new entry in the file. The machine value should be the hostname you’re setting the credentials for, which is github.com in this case. The login value should then be your GitHub username. Finally, the password value should be the GitHub personal access token you created.

      ~/.netrc

      machine github.com
      login your_github_username
      password your_github_access_token
      

      If you’d prefer, you can also put the entire entry on one line in the file as well:

      ~/.netrc

      machine github.com login your_github_username password your_github_access_token
      

      Note: If you are using Bitbucket for your source code hosting you may also need to add a second entry for api.bitbucket.org in addition to bitbucket.org. In the past, Bitbucket provided hosting for multiple types of version control, so Go would use the API to check the type of repository before trying to download it. While this is no longer the case, the API check still exists. If you encounter this issue, an example error message may look like this:

      go get bitbucket.org/your_github_username/mysecret: reading https://api.bitbucket.org/2.0/repositories/your_bitbucket_username/protocol?fields=scm: 403 Forbidden
          server response: Access denied. You must have write or admin access.
      

      If you see the 403 Forbidden error when trying to download a private module, double check the hostname Go is trying to connect to. It could indicate another hostname, such as api.bitbucket.org, that you need to add to your .netrc file.

      Now your environment is set up to use HTTPS authentication for downloading your private module. Even though HTTPS is the default way Go and Git will try to download a module, it’s also possible to tell Git to use SSH instead. Using SSH instead of HTTPS can be useful so you can use the same SSH key you used to push your private module. It also allows you to use deploy keys when setting up a CI/CD environment if you’d rather not create a personal access token.

      Providing Private Module Credentials for SSH

      To use your SSH key as the authentication method for your private Go module instead of HTTPS, Git provides a configuration option called insteadOf. The insteadOf option allows you to say that “instead of” using https://github.com/ as the request URL for all Git requests, you’d prefer to use ssh://[email protected]/.

      On Linux, MacOS, and WSL this configuration lives in the .gitconfig file. You may already be familiar with this file as it’s also where your commit email address and name are configured as well. To edit the file, use nano, or your favorite text editor, and open the ~/.gitconfig file in your home directory:

      Once you have the file open, edit it to include a url section for ssh://[email protected]/ as in the example below:

      ~/.gitconfig

      [user]
          email = [email protected]
          name = Sammy the Shark
      
      [url "ssh://[email protected]/"]
          insteadOf = https://github.com/
      

      The order of the url section relative to the user section doesn’t matter, and you also don’t need to worry if there’s nothing else in the file except for the url section you just added. The order of the email and name fields inside the user section also does not matter.

      This new section tells Git that any URL you use that starts with https://github.com/ should have that prefixed replaced with ssh://[email protected]/ instead. Since Go uses HTTPS by default, this also affects your go get commands. Using your private module as an example, this means Go turns the github.com/your_github_username/mysecret import path into the URL https://github.com/your_github_username/mysecret. When Git encounters this URL it will see the URL matches the https://github.com/ prefix referenced by insteadOf and will turn the resulting URL into ssh://[email protected]/your_github_username/mysecret.

      This same pattern can be used for domains other than GitHub as long as the ssh://git@ URL works for that host as well.

      In this section, you configured Git to use SSH to download Go modules by updating your .gitconfig file and adding a url section. Now that authentication for your private module is set up, you can access it for use in your Go programs.

      Using a Private Module

      In the previous sections, you configured Go to access your private Go module via HTTPS, SSH, or possibly both. Now that Go can access your private module, it can be used similar to any public module you may have used in the past. In this section, you’ll create a new Go module that uses your private module.

      In the directory you use for your projects, such as projects, create a directory named myproject for the new project using the mkdir command:

      Once the directory is created, go to the directory using cd and initialize a new Go module using go mod init for your project based on the repository URL your project would live at, such as github.com/your_github_username/myproject. If you don’t plan for your project to be pushed to any other repository the module name could be just myproject, or any other name, but it’s good practice to use full URLs since most modules being shared will need them.

      • cd myproject
      • go mod init github.com/your_github_username/myproject

      Output

      go: creating new go.mod: module github.com/your_github_username/myproject

      Now, create your project’s first code file by opening main.go with nano, or your favorite text editor:

      Inside the file, set up the initial main function you will call your private module from:

      projects/myproject/main.go

      package main
      
      import "fmt"
      
      func main() {
          fmt.Println("My new project!")
      }
      

      To run your project now and make sure everything is set up correctly, you can use the go run command and provide it the main.go file:

      Output

      My new project!

      Next, add your private module as a dependency of your new project using go get, similar to how you would for a public module:

      • go get github.com/your_github_username/mysecret

      The go tool will then download your private module’s code and add it as a dependency using a version string matching your latest commit hash and the time of that commit:

      Output

      go: downloading github.com/your_github_username/mysecret v0.0.0-20210920195630-bda059d63fa2 go get: added github.com/your_github_username/mysecret v0.0.0-20210920195630-bda059d63fa2

      Finally, open the main.go file again and update it to add a call to your private module’s SecretProcess function in the main function. You’ll also need to update the import statement to add your github.com/your_github_username/mysecret private module as an import as well:

      projects/myproject/main.go

      package main
      
      import (
          "fmt"
      
          "github.com/your_github_username/mysecret"
      )
      
      func main() {
          fmt.Println("My new project!")
          mysecret.SecretProcess()
      }
      

      To see the final project running with your private module, use the go run command again while providing the main.go file as the parameter:

      You will see the My new project! line from the original code, but now you’ll also see a Running the secret process! line from your imported mysecret module as well:

      Output

      My new project! Running the secret process!

      In this section, you used go init to create a new Go module to access the private module you published earlier. Once you had the module created, you then used go get to download your private module as you would with a public Go module. Finally, you used go run to compile and run your Go program using the private module.

      Conclusion

      In this tutorial, you created and published a private Go module. You also set up both HTTPS and SSH authentication to access your private Go module. Finally, you used your private module in a new project.

      For more information on Go modules, the Go project has a series of blog posts detailing how the Go tools interact with and understand modules. The Go project also has a very detailed and technical reference for Go modules in the Go Modules Reference.

      In addition to the GOPRIVATE environment variable, more variables are available to use when working with private Go modules. They can be seen in detail in the Private Modules section of the Go Modules Reference.

      If you’re interested in exploring the .netrc file in more detail, the GNU website on .netrc includes a list of all the available keywords. The git-config documentation also includes more information about how the insteadOf configuration option you used works in addition to other options that are available.

      This tutorial is also part of the DigitalOcean How to Code in Go series. The series covers a number of Go topics, from installing Go for the first time to how to use the language itself.



      Source link

      How To Work With Zip Files in Node.js


      The author selected Open Sourcing Mental Illness to receive a donation as part of the Write for DOnations program.

      Introduction

      Working with files is one of the common tasks among developers. As your files grow in size, they start taking significant space on your hard drive. Sooner or later you may need to transfer the files to other servers or upload multiple files from your local machine to different platforms. Some of these platforms have file size limits, and won’t accept large files. To get around this, you can group the files into a single ZIP file. A ZIP file is an archive format that packs and compresses files with the lossless compression algorithm. The algorithm can reconstruct the data without any data loss. In Node.js, you can use the adm-zip module to create and read ZIP archives.

      In this tutorial, you will use adm-zip module to compress, read, and decompress files. First, you’ll combine multiple files into a ZIP archive using adm-zip. You’ll then list the ZIP archive contents. After that, you’ll add a file to an existing ZIP archive, and then finally, you’ll extract a ZIP archive into a directory.

      Prerequisites

      To follow this tutorial, you’ll need:

      Step 1 — Setting Up the Project

      In this step, you’ll create the directory for your project and install adm-zip as a dependency. This directory is where you’ll keep your program files. You’ll also create another directory containing text files and an image. You’ll archive this directory in the next section.

      Create a directory called zip_app with the following command:

      Navigate into the newly created directory with the cd command:

      Inside the directory, create a package.json file to manage the project dependencies:

      The -y option creates a default package.json file.

      Next, install adm-zip with the npm install command:

      After you run the command, npm will install adm-zip and update the package.json file.

      Next, create a directory called test and move into it:

      In this directory, you will create three text files and download an image. The three files will be filled with dummy content to make their file sizes larger. This will help to demonstrate ZIP compression when you archive this directory.

      Create the file1.txt and fill it with dummy content using the following command:

      • yes "dummy content" | head -n 100000 > file1.txt

      The yes command logs the string dummy content repeatedly. Using the pipe command |, you send the output from the yes command to be used as input for the head command. The head command prints part of the given input into the standard output. The -n option specifies the number of lines that should be written to the standard output. Finally, you redirect the head output to a new file file1.txt using >.

      Create a second file with the string “dummy content” repeated 300,000 lines:

      • yes "dummy content" | head -n 300000 > file2.txt

      Create another file with the dummy content string repeated 600,000 lines:

      • yes "dummy content" | head -n 600000 > file3.txt

      Finally, download an image into the directory using curl:

      • curl -O https://assets.digitalocean.com/how-to-process-images-in-node-js-with-sharp/underwater.png

      Move back into the main project directory with the following command:

      The .. will move you to the parent directory, which is zip_app.

      You’ve now created the project directory, installed adm-zip, and created a directory with files for archiving. In the next step, you’ll archive a directory using the adm-zip module.

      Step 2 — Creating a ZIP Archive

      In this step, you’ll use adm-zip to compress and archive the directory you created in the previous section.

      To archive the directory, you’ll import the adm-zip module and use the module’s addLocalFolder() method to add the directory to the adm-zip module’s ZIP object. Afterward, you’ll use the module’s writeZip() method to save the archive in your local system.

      Create and open a new file createArchive.js in your preferred text editor. This tutorial uses nano, a command-line text editor:

      Next, require in the adm-zip module in your createArchive.js file:

      zip_app/createArchive.js

      const AdmZip = require("adm-zip");
      

      The adm-zip module provides a class that contains methods for creating ZIP archives.

      Since it’s common to encounter large files during the archiving process, you might end up blocking the main thread until the ZIP archive is saved. To write non-blocking code, you’ll define an asynchronous function to create and save a ZIP archive.

      In your createArchive.js file, add the following highlighted code:

      zip_app/createArchive.js

      
      const AdmZip = require("adm-zip");
      
      async function createZipArchive() {
        const zip = new AdmZip();
        const outputFile = "test.zip";
        zip.addLocalFolder("./test");
        zip.writeZip(outputFile);
        console.log(`Created ${outputFile} successfully`);
      }
      
      createZipArchive();
      

      createZipArchive is an asynchronous function that creates a ZIP archive from a given directory. What makes it asynchronous is the async keyword you defined before the function label. Within the function, you create an instance of the adm-zip module, which provides methods you can use for reading and creating archives. When you create an instance, adm-zip creates an in-memory ZIP where you can add files or directories.

      Next, you define the archive name and store it in the outputDir variable. To add the test directory to the in-memory archive, you invoke the addLocalFolder() method from adm-zip with the directory path as an argument.

      After the directory is added, you invoke the writeZip() method from adm-zip with a variable containing the name of the ZIP archive. The writeZip() method saves the archive to your local disk.

      Once that’s done, you invoke console.log() to log that the ZIP file has been created successfully.

      Finally, you call the createZipArchive() function.

      Before you run the file, wrap the code in a try…catch block to handle runtime errors:

      zip_app/createArchive.js

      const AdmZip = require("adm-zip");
      
      async function createZipArchive() {
        try {
          const zip = new AdmZip();
          const outputFile = "test.zip";
          zip.addLocalFolder("./test");
          zip.writeZip(outputFile);
          console.log(`Created ${outputFile} successfully`);
        } catch (e) {
          console.log(`Something went wrong. ${e}`);
        }
      }
      
      createZipArchive();
      

      Within the try block, the code will attempt to create a ZIP archive. If successful, the createZipArchive() function will exit, skipping the catch block. If creating a ZIP archive triggers an error, execution will skip to the catch block and log the error in the console.

      Save and exit the file in nano with CTRL+X. Enter y to save the changes, and confirm the file by pressing ENTER on Windows, or the RETURN key on the Mac.

      Run the createArchive.js file using the node command:

      You’ll receive the following output:

      Output

      Created test.zip successfully

      List the directory contents to see if the ZIP archive has been created:

      You’ll receive the following output showing the archive among the contents:

      Output

      createArchive.js node_modules package-lock.json package.json test test.zip

      With the confirmation that the ZIP archive has been created, you’ll compare the ZIP archive, and the test directory file size to see if the compression works.

      Check the test directory size using the du command:

      The -h flag instructs du to show the directory size in a human-readable format.

      After running the command, you will receive the following output:

      Output

      15M test

      Next, check the test.zip archive file size:

      The du command logs the following output:

      Output

      760K test.zip

      As you can see, creating the ZIP file has dropped the directory size from 15 Megabytes(MB) to 760 Kilobytes(KB), which is a huge difference. The ZIP file is more portable and smaller in size.

      Now that you created a ZIP archive, you’re ready to list the contents in a ZIP file.

      Step 3 — Listing Files in a ZIP Archive

      In this step, you’ll read and list all files in a ZIP archive using adm-zip. To do that, you’ll instantiate the adm-zip module with your ZIP archive path. You’ll then call the module’s getEntries() method which returns an array of objects. Each object holds important information about an item in the ZIP archive. To list the files, you’ll iterate over the array and access the filename from the object, and log it in the console.

      Create and open readArchive.js in your favorite text editor:

      In your readArchive.js, add the following code to read and list contents in a ZIP archive:

      zip_app/readArchive.js

      const AdmZip = require("adm-zip");
      
      async function readZipArchive(filepath) {
        try {
          const zip = new AdmZip(filepath);
      
          for (const zipEntry of zip.getEntries()) {
            console.log(zipEntry.toString());
          }
        } catch (e) {
          console.log(`Something went wrong. ${e}`);
        }
      }
      
      readZipArchive("./test.zip");
      

      First, you require in the adm-zip module.

      Next, you define the readZipArchive() function, which is an asynchronous function. Within the function, you create an instance of adm-zip with the path of the ZIP file you want to read. The file path is provided by the filepath parameter. adm-zip will read the file and parse it.

      After reading the archive, you define a for....of statement that iterates over objects in an array that the getEntries() method from adm-zip returns when invoked. On each iteration, the object is assigned to the zipEntry variable. Inside the loop, you convert the object into a string that represents the object using the Node.js toString() method, then log it in the console using the console.log() method.

      Finally, you invoke the readZipArchive() function with the ZIP archive file path as an argument.

      Save and exit your file, then run the file with the following command:

      You will get output that resembles the following(edited for brevity):

      Output

      { "entryName": "file1.txt", "name": "file1.txt", "comment": "", "isDirectory": false, "header": { ... }, "compressedData": "<27547 bytes buffer>", "data": "<null>" } ...

      The console will log four objects. The other objects have been edited out to keep the tutorial brief.

      Each file in the archive is represented with an object similar to the one in the preceding output. To get the filename for each file, you need to access the name property.

      In your readArchive.js file, add the following highlighted code to access each filename:

      zip_app/readArchive.js

      const AdmZip = require("adm-zip");
      
      async function readZipArchive(filepath) {
        try {
          const zip = new AdmZip(filepath);
      
          for (const zipEntry of zip.getEntries()) {
            console.log(zipEntry.name);
          }
        } catch (e) {
          console.log(`Something went wrong. ${e}`);
        }
      }
      
      readZipArchive("./test.zip");
      

      Save and exit your text editor. Now, run the file again with the node command:

      Running the file results in the following output:

      Output

      file1.txt file2.txt file3.txt underwater.png

      The output now logs the filename of each file in the ZIP archive.

      You can now read and list each file in a ZIP archive. In the next section, you’ll add a file to an existing ZIP archive.

      Step 4 — Adding a File to an Existing Archive

      In this step, you’ll create a file and add it to the ZIP archive you created earlier without extracting it. First, you’ll read the ZIP archive by creating an adm-zip instance. Second, you’ll invoke the module’s addFile() method to add the file in the ZIP. Finally, you’ll save the ZIP archive in the local system.

      Create another file file4.txt with dummy content repeated 600,000 lines:

      • yes "dummy content" | head -n 600000 > file4.txt

      Create and open updateArchive.js in your text editor:

      Require in the adm-zip module and the fs module that allows you to work with files in your updateArchive.js file:

      const AdmZip = require("adm-zip");
      const fs = require("fs").promises;
      

      You require in the promise-based version of the fs module version, which allows you to write asynchronous code. When you invoke an fs method, it will return a promise.

      Next in your updateArchive.js file, add the following highlighted code to add a new file to the ZIP archive:

      zip_app/updateArchive.js

      const AdmZip = require("adm-zip");
      const fs = require("fs").promises;
      
      async function updateZipArchive(filepath) {
        try {
          const zip = new AdmZip(filepath);
      
          content = await fs.readFile("./file4.txt");
          zip.addFile("file4.txt", content);
          zip.writeZip(filepath);
          console.log(`Updated ${filepath} successfully`);
        } catch (e) {
          console.log(`Something went wrong. ${e}`);
        }
      }
      
      updateZipArchive("./test.zip");
      

      updateZipArchive is an asynchronous function that reads a file in the filesystem and adds it to an existing ZIP. In the function, you create an instance of adm-zip with the ZIP archive file path in the filepath as a parameter. Next, you invoke the fs module’s readFile() method to read the file in the file system. The readFile() method returns a promise, which you resolve with the await keyword (await is valid in only asynchronous functions). Once resolved, the method returns a buffer object, which contains the file contents.

      Next, you invoke the addFile() method from adm-zip. The method takes two arguments. The first argument is the filename you want to add to the archive, and the second argument is the buffer object containing the contents of the file that the readFile() method reads.

      Afterwards, you invoke adm-zip module’s writeZip() method to save and write new changes in the ZIP archive. Once that’s done, you call the console.log() method to log a success message.

      Finally, you invoke the updateZipArchive() function with the Zip archive file path as an argument.

      Save and exit your file. Run the updateArchive.js file with the following command:

      You’ll see output like this:

      Output

      Updated ./test.zip successfully

      Now, confirm that the ZIP archive contains the new file. Run the readArchive.js file to list the contents in the ZIP archive with the following command:

      You’ll receive the following output:

      file1.txt
      file2.txt
      file3.txt
      file4.txt
      underwater.png
      

      This confirms that the file has been added to the ZIP.

      Now that you can add a file to an existing archive, you’ll extract the archive in the next section.

      In this step, you’ll read and extract all contents in a ZIP archive into a directory. To extract a ZIP archive, you’ll instantiate adm-zip with the archive file path. After that, you’ll invoke the module’s extractAllTo() method with the directory name you want your extracted ZIP contents to reside.

      Create and open extractArchive.js in your text editor:

      Require in the adm-zip module and the path module in your extractArchive.js file:

      zip_app/extractArchive.js

      const AdmZip = require("adm-zip");
      const path = require("path");
      

      The path module provides helpful methods for dealing with file paths.

      Still in your extractArchive.js file, add the following highlighted code to extract an archive:

      zip_app/extractArchive.js

      const AdmZip = require("adm-zip");
      const path = require("path");
      
      async function extractArchive(filepath) {
        try {
          const zip = new AdmZip(filepath);
          const outputDir = `${path.parse(filepath).name}_extracted`;
          zip.extractAllTo(outputDir);
      
          console.log(`Extracted to "${outputDir}" successfully`);
        } catch (e) {
          console.log(`Something went wrong. ${e}`);
        }
      }
      
      extractArchive("./test.zip");
      

      extractArchive() is an asynchronous function that takes a parameter containing the file path of the ZIP archive. Within the function, you instantiate adm-zip with the ZIP archive file path provided by the filepath parameter.

      Next, you define a template literal. Inside the template literal placeholder (${}), you invoke the parse() method from the path module with the file path. The parse() method returns an object. To get the name of the ZIP file without the file extension, you append the name property to the object that the parse() method returns. Once the archive name is returned, the template literal interpolates the value with the _extracted string. The value is then stored in the outputDir variable. This will be the name of the extracted directory.

      Next, you invoke adm-zip module’s extractAllTo method with the directory name stored in the outputDir to extract the contents in the directory. After that, you invoke console.log() to log a success message.

      Finally, you call the extractArchive() function with the ZIP archive path.

      Save your file and exit the editor, then run the extractArchive.js file with the following command:

      You receive the following output:

      Output

      Extracted to "test_extracted" successfully

      Confirm that the directory containing the ZIP contents has been created:

      You will receive the following output:

      Output

      createArchive.js file4.txt package-lock.json readArchive.js test.zip updateArchive.js extractArchive.js node_modules package.json test test_extracted

      Now, navigate into the directory containing the extracted contents:

      List the contents in the directory:

      You will receive the following output:

      Output

      file1.txt file2.txt file3.txt file4.txt underwater.png

      You can now see that the directory has all the files that were in the original directory.

      You’ve now extracted the ZIP archive contents into a directory.

      Conclusion

      In this tutorial, you created a ZIP archive, listed its contents, added a new file to the archive, and extracted all of its content into a directory using adm-zip module. This will serve as a good foundation for working with ZIP archives in Node.js.

      To learn more about adm-zip module, view the adm-zip documentation. To continue building your Node.js knowledge, see How To Code in Node.js series



      Source link

      How To Configure Suricata as an Intrusion Prevention System (IPS) on Rocky Linux 8


      Not using Rocky Linux 8?


      Choose a different version or distribution.

      Introduction

      In this tutorial you will learn how to configure Suricata’s built-in Intrusion Prevention System (IPS) mode on Rocky Linux 8. By default Suricata is configured to run as an Intrusion Detection System (IDS), which only generates alerts and logs suspicious traffic. When you enable IPS mode, Suricata can actively drop suspicious network traffic in addition to generating alerts for further analysis.

      Before enabling IPS mode, it is important to check which signatures you have enabled, and their default actions. An incorrectly configured signature, or a signature that is overly broad may result in dropping legitimate traffic to your network, or even block you from accessing your servers over SSH and other management protocols.

      In the first part of this tutorial you will check the signatures that you have installed and enabled. You will also learn how to include your own signatures. Once you know which signatures you would like to use in IPS mode, you’ll convert their default action to drop or reject traffic. With your signatures in place, you’ll learn how to send network traffic through Suricata using the netfilter NFQUEUE iptables target, and then generate some invalid network traffic to ensure that Suricata drops it as expected.

      Prerequisites

      If you have been following this tutorial series then you should already have Suricata running on a Rocky Linux 8 server.

      • If you still need to install Suricata then you can follow How To Install Suricata on Rocky Linux 8

      • You should also have the ET Open Ruleset downloaded using the suricata-update command, and included in your Suricata signatures.

      • The jq command line JSON processing tool. If you do not have it installed from a previous tutorial, you can do so using the dnf command:

      You may also have custom signatures that you would like to use from the previous Understanding Suricata Signatures tutorial.

      Step 1 — Including Custom Signatures

      The previous tutorials in this series explored how to install and configure Suricata, as well as how to understand signatures. If you would like to create and include your own rules then you need to edit Suricata’s /etc/suricata/suricata.yaml file to include a custom path to your signatures.

      First, let’s find your server’s public IPs so that you can use them in your custom signatures. To find your IPs you can use the ip command:

      You should receive output like the following:

      Output

      lo UNKNOWN 127.0.0.1/8 ::1/128 eth0 UP 203.0.113.0.5/20 10.20.0.5/16 2001:DB8::1/32 fe80::94ad:d4ff:fef9:cee0/64 eth1 UP 10.137.0.2/16 fe80::44a2:ebff:fe91:5187/64

      Your public IP address(es) will be similar to the highlighted 203.0.113.0.5 and 2001:DB8::1/32 IPs in the output.

      Now let’s create the following custom signature to scan for SSH traffic to non-SSH ports and include it in a file called /var/lib/suricata/rules/local.rules. Open the file with nano or your preferred editor:

      • sudo vi /var/lib/suricata/rules/local.rules

      Copy and paste the following signature:

      Invalid SSH Traffic Signature

      alert ssh any any -> 203.0.113.0.5 !22 (msg:"SSH TRAFFIC on non-SSH port"; flow:to_client, not_established; classtype: misc-attack; target: dest_ip; sid:1000000;)
      alert ssh any any -> 2001:DB8::1/32 !22 (msg:"SSH TRAFFIC on non-SSH port"; flow:to_client, not_established; classtype: misc-attack; target: dest_ip; sid:1000001;)
      

      Substitute your server’s public IP address in place of the 203.0.113.5 and 2001:DB8::1/32 addresses in the rule. If you are not using IPv6 then you can skip adding that signature in this and the following rules.

      You can continue adding custom signatures to this local.rules file depending on your network and applications. For example, if you wanted to alert about HTTP traffic to non-standard ports, you could use the following signatures:

      HTTP traffic on non-standard port signature

      alert http any any -> 203.0.113.0.5 !80 (msg:"HTTP REQUEST on non-HTTP port"; flow:to_client, not_established; classtype:misc-activity; sid:1000002;)
      alert http any any -> 2001:DB8::1/32 !80 (msg:"HTTP REQUEST on non-HTTP port"; flow:to_client, not_established; classtype:misc-activity; sid:1000003;)
      

      To add a signature that checks for TLS traffic to ports other than the default 443 for web servers, add the following:

      TLS traffic on non-standard port signature

      alert tls any any -> 203.0.113.0.5 !443 (msg:"TLS TRAFFIC on non-TLS HTTP port"; flow:to_client, not_established; classtype:misc-activity; sid:1000004;)
      alert tls any any -> 2001:DB8::1/32 !443 (msg:"TLS TRAFFIC on non-TLS HTTP port"; flow:to_client, not_established; classtype:misc-activity; sid:1000005;)
      

      When you are done adding signatures, save and close the file. If you are using vi, press ESC and then :x then ENTER to save and exit.

      Now that you have some custom signatures defined, edit Suricata’s /etc/suricata/suricata.yaml configuration file using nano or your preferred editor to include them:

      • sudo vi /etc/suricata/suricata.yaml

      Find the rule-files: portion of the configuration. If you are using vi enter 1879gg to go to the line. The exact location in your file may be different, but you should be in the correct general region of the file.

      Edit the section and add the following highlighted - local.rules line:

      /etc/suricata/suricata.yaml

      . . .
      rule-files:
        - suricata.rules
        - local.rules
      . . .
      

      Save and exit the file. Be sure to validate Suricata’s configuration after adding your rules. To do so run the following command:

      • sudo suricata -T -c /etc/suricata/suricata.yaml -v

      The test can take some time depending on how many rules you have loaded in the default suricata.rules file. If you find the test takes too long, you can comment out the - suricata.rules line in the configuration by adding a # to the beginning of the line and then run your configuration test again. Be sure to remove the # comment if you plan to use the suricata.rules signature in your final running configuration.

      Once you are satisfied with the signatures that you have created or included using the suricata-update tool, you can proceed to the next step, where you’ll switch the default action for your signatures from alert or log to actively dropping traffic.

      Step 2 — Configuring Signature Actions

      Now that you have your custom signatures tested and working with Suricata, you can change the action to drop or reject. When Suricata is operating in IPS mode, these actions will actively block invalid traffic for any matching signature.

      These two actions are described in the previous tutorial in this series, Understanding Suricata Signatures. The choice of which action to use is up to you. A drop action will immediately discard a packet and any subsequent packets that belong to the network flow. A reject action will send both the client and server a reset packet if the traffic is TCP-based, and an ICMP error packet for any other protocol.

      Let’s use the custom rules from the previous section and convert them to use the drop action, since the traffic that they match is likely to be a network scan, or some other invalid connection.

      Open your /var/lib/suricata/rules/local.rules file using nano or your preferred editor and change the alert action at the beginning of each line in the file to drop:

      • sudo vi /var/lib/suricata/rules/local.rules

      /var/lib/suricata/rules/local.rules

      drop ssh any any -> 203.0.113.0.5 !22 (msg:"SSH TRAFFIC on non-SSH port"; classtype: misc-attack; target: dest_ip; sid:1000000;)
      drop ssh any any -> 2001:DB8::1/32 !22 (msg:"SSH TRAFFIC on non-SSH port"; classtype: misc-attack; target: dest_ip; sid:1000001;)
      . . .
      

      Repeat the step above for any signatures in /var/lib/suricata/rules/suricata.rules that you would like to convert to drop or reject mode.

      Note: If you ran suricata-update in the prerequisite tutorial, you may have more than 30,000 signatures included in your suricata.rules file.

      If you convert every signature to drop or reject you risk blocking legitimate access to your network or servers. Instead, leave the rules in suricata.rules for the time being, and add your custom signatures to local.rules. Suricata will continue to generate alerts for suspicious traffic that is described by the signatures in suricata.rules while it is running in IPS mode.

      After you have a few days or weeks of alerts collected, you can analyze them and choose the relevant signatures to convert to drop or reject based on their sid.

      Once you have all the signatures configured with the action that you would like them to take, the next step is to reconfigure and then restart Suricata in IPS mode.

      Step 3 — Enabling nfqueue Mode

      Suricata runs in IDS mode by default, which means it will not actively block network traffic. To switch to IPS mode, you’ll need to edit Suricata’s /etc/sysconfig/suricata configuration file.

      Open the file in nano or your preferred editor:

      • sudo vi /etc/sysconfig/suricata

      Find the OPTIONS="-i eth0 --user suricata" line and comment it out by adding a # to the beginning of the line. Then add a new line OPTIONS="-q 0 -vvv --user suricata" line that tells Suricata to run in IPS mode.

      Your file should have the following highlighted lines in it when you are done editing:

      /etc/sysconfig/suricata

      . . .
      # OPTIONS="-i eth0 --user suricata"
      OPTIONS="-q 0 -vvv --user suricata"
      . . .
      

      Save and close the file. Now you can restart Suricata using systemctl:

      • sudo systemctl restart suricata.service

      Check Suricata’s status using systemctl:

      • sudo systemctl status suricata.service

      You should receive output like the following:

      Output

      ● suricata.service - Suricata Intrusion Detection Service Loaded: loaded (/usr/lib/systemd/system/suricata.service; disabled; vendor preset: disabled) Active: active (running) since Tue 2021-12-14 16:52:07 UTC; 6s ago Docs: man:suricata(1) Process: 44256 ExecStartPre=/bin/rm -f /var/run/suricata.pid (code=exited, status=0/SUCCESS) Main PID: 44258 (Suricata-Main) Tasks: 10 (limit: 11188) Memory: 52.8M CGroup: /system.slice/suricata.service └─44258 /sbin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid -q 0 -vvv --user suricata . . . Dec 14 16:52:07 suricata suricata[44258]: 14/12/2021 -- 16:52:07 - <Notice> - all 4 packet processing threads, 4 management threads initialized, engine started.

      Note the highlighted active (running) line that indicates Suricata restarted successfully.

      With this change you are now ready to send traffic to Suricata using Firewalld in the next step.

      Step 4 — Configuring Firewalld To Send Traffic to Suricata

      Now that you have configured Suricata to process traffic in IPS mode, the next step is to direct incoming packets to Suricata. If you followed the prerequisite tutorials for this series and are using a Rocky Linux 8 system, you should have Firewalld installed and enabled.

      To add the required rules for Suricata to Firewalld, you will need to run the following commands:

      • sudo firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 0 -p tcp --dport 22 -j NFQUEUE --queue-bypass
      • sudo firewall-cmd --permanent --direct --add-rule ipv4 filter OUTPUT 0 -p tcp --sport 22 -j NFQUEUE --queue-bypass

      These two rules ensure that SSH traffic on IPv4 interfaces will bypass Suricata so that you can connect to your server using SSH, even when Suricata is not running. Without these rules, an incorrect or overly broad signature could block your SSH access. Additionally, if Suricata is stopped, all traffic will be sent to the NFQUEUE target and then dropped since Suricata is not running.

      Add the same rules for IPv6 using the following commands:

      • sudo firewall-cmd --permanent --direct --add-rule ipv6 filter INPUT 0 -p tcp --dport 22 -j NFQUEUE --queue-bypass
      • sudo firewall-cmd --permanent --direct --add-rule ipv6 filter OUTPUT 0 -p tcp --sport 22 -j NFQUEUE --queue-bypass

      Next, add FORWARD rules to ensure that if your server is acting as a gateway for other systems, all that traffic will also go to Suricata for processing.

      • sudo firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 0 -j NFQUEUE
      • sudo firewall-cmd --permanent --direct --add-rule ipv6 filter FORWARD 0 -j NFQUEUE

      The final two INPUT and OUTPUT rules send all remaining traffic that is not SSH traffic to Suricata for processing.

      • sudo firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 1 -j NFQUEUE
      • sudo firewall-cmd --permanent --direct --add-rule ipv4 filter OUTPUT 1 -j NFQUEUE

      Repeat the commands for IPv6 traffic:

      • sudo firewall-cmd --permanent --direct --add-rule ipv6 filter INPUT 1 -j NFQUEUE
      • sudo firewall-cmd --permanent --direct --add-rule ipv6 filter OUTPUT 1 -j NFQUEUE

      Now reload Firewalld to make the rules persistent:

      • sudo firewall-cmd --reload

      Note: If you are using another firewall like iptables you will need to modify these rules to match the format your firewall expects.

      At this point in the tutorial you have Suricata configured to run in IPS mode, and your network traffic is being sent to Suricata by default. You will be able to restart your server at any time and your Suricata and firewall rules will be persistent.

      The last step in this tutorial is to verify Suricata is dropping traffic correctly.

      Step 5 — Testing Invalid Traffic

      Now that you have Suricata and your firewall configured to process network traffic, you can test whether Suricata will drop packets that match your custom and other included signatures.

      Recall signature sid:2100498 from the previous tutorial, which is modified in this example to drop matching packets:

      sid:2100498

      drop ip any any -> any any (msg:"GPL ATTACK_RESPONSE id check returned root"; content:"uid=0|28|root|29|"; classtype:bad-unknown; sid:2100498; rev:7; metadata:created_at 2010_09_23, updated_at 2010_09_23;)
      

      Find and edit the rule in your /var/lib/suricata/rules/suricata.rules file to use the drop action if you have the signature included there. Otherwise, add the rule to your /var/lib/suricata/rules/local.rules file.

      Send Suricata the SIGUSR2 signal to get it to reload its signatures:

      • sudo kill -usr2 $(pidof suricata)

      Now test the rule using curl:

      • curl --max-time 5 http://testmynids.org/uid/index.html

      You should receive an error stating that the request timed out, which indicates Suricata blocked the HTTP response:

      Output

      curl: (28) Operation timed out after 5000 milliseconds with 0 out of 39 bytes received

      You can confirm that Suricata dropped the HTTP response using jq to examine the eve.log file:

      • sudo jq 'select(.alert .signature_id==2100498)' /var/log/suricata/eve.json

      You should receive output like the following:

      Output

      { . . . "community_id": "1:SbOgFh2T3DZvwsoyMH4xfxOoVas=", "alert": { "action": "blocked", "gid": 1, "signature_id": 2100498, "rev": 7, "signature": "GPL ATTACK_RESPONSE id check returned root", "category": "Potentially Bad Traffic", "severity": 2, "metadata": { "created_at": [ "2010_09_23" ], "updated_at": [ "2010_09_23" ] } }, "http": { "hostname": "testmynids.org", "url": "/uid/index.html", "http_user_agent": "curl/7.61.1", "http_content_type": "text/html", "http_method": "GET", "protocol": "HTTP/1.1", "status": 200, "length": 39 }, . . .

      The highlighted "action": "blocked" line confirms that the signature matched, and Suricata dropped or rejected the test HTTP request.

      Conclusion

      In this tutorial you configured Suricata to block suspicious network traffic using its built-in IPS mode on Rocky Linux 8. You also added custom signatures to examine and block SSH, HTTP, and TLS traffic on non-standard ports. To tie everything together, you also added firewall rules to direct traffic through Suricata for processing.

      Now that you have Suricata installed and configured in IPS mode, and can write your own signatures that either alert on or drop suspicious traffic, you can continue monitoring your servers and networks, and refining your signatures.

      Once you are satisfied with your Suricata signatures and configuration, you can continue with the last tutorial in this series, which will guide you through sending logs from Suricata to a Security and Information Event Management (SIEM) system built using the Elastic Stack.



      Source link