One place for hosting & domains

      Private

      How to Set Up Squid Proxy for Private Connections on Ubuntu 20.04


      Introduction

      Proxy servers are a type of server application that functions as a gateway between an end user and an internet resource. Through a proxy server, an end user is able to control and monitor their web traffic for a wide variety of purposes, including privacy, security, and caching. For example, you can use a proxy server to make web requests from a different IP address than your own. You can also use a proxy server to research how the web is served differently from one jurisdiction to the next, or avoid some methods of surveillance or web traffic throttling.

      Squid is a stable, popular, open-source HTTP proxy. In this tutorial, you will be installing and configuring Squid to provide an HTTP proxy on a Ubuntu 20.04 server.

      Prerequisites

      To complete this guide, you will need:

      You will use the domain name your_domain in this tutorial, but you should substitute this with your own domain name, or IP address.

      Step 1 — Installing Squid Proxy

      Squid has many use cases beyond routing an individual user’s outbound traffic. In the context of large-scale server deployments, it can be used as a distributed caching mechanism, a load balancer, or another component of a routing stack. However, some methods of horizontally scaling server traffic that would typically have involved a proxy server have been surpassed in popularity by containerization frameworks such as Kubernetes, which distribute more components of an application. At the same time, using proxy servers to redirect web requests as an individual user has become increasingly popular for protecting your privacy. This is helpful to keep in mind when working with open-source proxy servers which may appear to have many dozens of features in a lower-priority maintenance mode. The use cases for a proxy have changed over time, but the fundamental technology has not.

      Begin by running the following commands as a non-root user to update your package listings and install Squid Proxy:

      • sudo apt update
      • sudo apt install squid

      Squid will automatically set up a background service and start after being installed. You can check that the service is running properly:

      • systemctl status squid.service

      Output

      ● squid.service - Squid Web Proxy Server Loaded: loaded (/lib/systemd/system/squid.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2021-12-15 21:45:15 UTC; 2min 11s ago

      By default, Squid does not allow any clients to connect to it from outside of this server. In order to enable that, you’ll need to make some changes to its configuration file, which is stored in /etc/squid/squid.conf. Open it in nano or your favorite text editor:

      • sudo nano /etc/squid/squid.conf

      Be advised that Squid’s default configuration file is very, very long, and contains a massive number of options that have been temporarily disabled by putting a # at the start of the line they’re on, also called being commented out. You will most likely want to search through the file to find the lines you want to edit. In nano, this is done by pressing Ctrl+W, entering your search term, pressing Enter, and then repeatedly pressing Alt+W to find the next instance of that term if needed.

      Begin by navigating to the line containing the phrase http_access deny all. You should see a block of text explaining Squid’s default access rules:

      /etc/squid/squid.conf

      . . . 
      #
      # INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
      #
      include /etc/squid/conf.d/*
      # Example rule allowing access from your local networks.
      # Adapt localnet in the ACL section to list your (internal) IP networks
      # from where browsing should be allowed
      #http_access allow localnet
      http_access allow localhost
      
      # And finally deny all other access to this proxy
      http_access deny all
      . . . 
      

      From this, you can see the current behavior – localhost is allowed; other connections are not. Note that these rules are parsed sequentially, so it’s a good idea to keep the deny all rule at the bottom of this configuration block. You could change that rule to allow all, enabling anyone to connect to your proxy server, but you probably don’t want to do that. Instead, you can add a line above http_access allow localhost that includes your own IP address, like so:

      /etc/squid/squid.conf

      #
      # INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
      #
      include /etc/squid/conf.d/*
      # Example rule allowing access from your local networks.
      acl localnet src your_ip_address
      # Adapt localnet in the ACL section to list your (internal) IP networks
      # from where browsing should be allowed
      #http_access allow localnet
      http_access allow localhost
      
      • acl means an Access Control List, a common term for permissions policies
      • localnet in this case is the name of your ACL.
      • src is where the request would originate from under this ACL, i.e., your IP address.

      If you don’t know your local IP address, it’s quickest to go to a site like What’s my IP which can tell you where you accessed it from. After making that change, save and close the file. If you are using nano, press Ctrl+X, and then when prompted, Y and then Enter.

      At this point, you could restart Squid and connect to it, but there’s more you can do in order to secure it first.

      Step 2 — Securing Squid

      Most proxies, and most client-side apps that connect to proxies (e.g., web browsers) support multiple methods of authentication. These can include shared keys, or separate authentication servers, but most commonly entail regular username-password pairs. Squid allows you to create username-password pairs using built-in Linux functionality, as an additional or an alternative step to restricting access to your proxy by IP address. To do that, you’ll create a file called /etc/squid/passwords and point Squid’s configuration to it.

      First, you’ll need to install some utilities from the Apache project in order to have access to a password generator that Squid likes.

      • sudo apt install apache2-utils

      This package provides the htpasswd command, which you can use in order to generate a password for a new Squid user. Squid’s usernames won’t overlap with system usernames in any way, so you can use the same name you’ve logged in with if you want. You’ll be prompted to add a password as well:

      • sudo htpasswd -c /etc/squid/passwords your_squid_username

      This will store your username along with a hash of your new password in /etc/squid/passwords, which will be used as an authentication source by Squid. You can cat the file afterward to see what that looks like:

      • sudo cat /etc/squid/passwords

      Output

      sammy:$apr1$Dgl.Mtnd$vdqLYjBGdtoWA47w4q1Td.

      After verifying that your username and password have been stored, you can update Squid’s configuration to use your new /etc/squid/passwords file. Using nano or your favorite text editor, reopen the Squid configuration file and add the following highlighted lines:

      • sudo nano /etc/squid/squid.conf

      /etc/squid/squid.conf

      …
      #
      # INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
      #
      include /etc/squid/conf.d/*
      auth_param basic program /usr/lib/squid3/basic_ncsa_auth /etc/squid/passwords
      auth_param basic realm proxy
      acl authenticated proxy_auth REQUIRED
      # Example rule allowing access from your local networks.
      acl localnet src your_ip_address
      # Adapt localnet in the ACL section to list your (internal) IP networks
      # from where browsing should be allowed
      #http_access allow localnet
      http_access allow localhost
      http_access allow authenticated
      # And finally deny all other access to this proxy
      http_access deny all
      …
      

      These additional directives tell Squid to check in your new passwords file for password hashes that can be parsed using the basic_ncsa_auth mechanism, and to require authentication for access to your proxy. You can review Squid’s documentation for more information on this or other authentication methods. After that, you can finally restart Squid with your configuration changes. This might take a moment to complete.

      • sudo systemctl restart squid.service

      And don’t forget to open port 3128 in your firewall if you’re using ufw:

      In the next step, you’ll connect to your proxy at last.

      Step 3 — Connecting through Squid

      In order to demonstrate your Squid server, you’ll use a command line program called curl, which is popular for making different types of web requests. In general, if you want to verify whether a given connection should be working in a browser under ideal circumstances, you should always test first with curl. You’ll be using curl on your local machine in order to do this – it’s installed by default on all modern Windows, Mac, and Linux environments, so you can open any local shell to run this command:

      • curl -v -x http://your_squid_username:your_squid_password@your_server_ip:3128 http://www.google.com/

      The -x argument passes a proxy server to curl, and in this case you’re using the http:// protocol this time, specifying your username and password to this server, and then connecting to a known-working website like google.com. If the command was successful, you should see the following output:

      Output

      * Trying 138.197.103.77... * TCP_NODELAY set * Connected to 138.197.103.77 (138.197.103.77) port 3128 (#0) * Proxy auth using Basic with user 'sammy' > GET http://www.google.com/ HTTP/1.1

      It is also possible to access https:// websites with your Squid proxy without making any further configuration changes. These make use of a separate proxy directive called CONNECT in order to preserve SSL between the client and the server:

      • curl -v -x http://your_squid_username:your_squid_password@your_server_ip:3128 https://www.google.com/

      Output

      * Trying 138.197.103.77... * TCP_NODELAY set * Connected to 138.197.103.77 (138.197.103.77) port 3128 (#0) * allocate connect buffer! * Establish HTTP proxy tunnel to www.google.com:443 * Proxy auth using Basic with user 'sammy' > CONNECT www.google.com:443 HTTP/1.1 > Host: www.google.com:443 > Proxy-Authorization: Basic c2FtbXk6c2FtbXk= > User-Agent: curl/7.55.1 > Proxy-Connection: Keep-Alive > < HTTP/1.1 200 Connection established < * Proxy replied OK to CONNECT request * CONNECT phase completed!

      The credentials that you used for curl should now work anywhere else you might want to use your new proxy server.

      Conclusion

      In this tutorial, you learned to deploy a popular, open-source API endpoint for proxying traffic with little to no overhead. Many applications have built-in proxy support (often at the OS level) going back decades, making this proxy stack highly reusable.

      Next, you may want to learn how to deploy Dante, a SOCKS proxy which can run alongside Squid for proxying different types of web traffic.

      Because one of the most common use cases for proxy servers is proxying traffic to and from different global regions, you may want to review how to use Ansible to automate server deployments next, in case you find yourself wanting to duplicate this configuration in other data centers.



      Source link

      How to Set Up Dante Proxy for Private Connections on Ubuntu 20.04


      Introduction

      Proxy servers are a type of server application that functions as a gateway between an end user and an internet resource. Through a proxy server, an end user is able to control and monitor their web traffic for a wide variety of purposes, including privacy, security, and caching. For example, you can use a proxy server to make web requests from a different IP address than your own. You can also use a proxy server to research how the web is served differently from one jurisdiction to the next, or avoid some methods of surveillance or web traffic throttling.

      Dante is a stable, popular, open-source SOCKS proxy. In this tutorial, you will be installing and configuring Dante to provide a SOCKS proxy on a Ubuntu 20.04 server.

      Prerequisites

      To complete this guide, you will need:

      You will use the domain name your_domain in this tutorial, but you should substitute this with your own domain name, or IP address.

      Step 1 — Installing Dante

      Dante is an open-source SOCKS proxy server. SOCKS is a less widely used protocol, but it is more efficient for some peer-to-peer applications, and is preferred over HTTP for some kinds of traffic. Begin by running the following commands as a non-root user to update your package listings and install Dante:

      • sudo apt update
      • sudo apt install dante-server

      Dante will also automatically set up a background service and start after being installed. However, it is designed to gracefully quit with an error message the first time it runs, because it ships with all of its features disabled. You can verify this by using the systemctl command:

      • systemctl status danted.service

      Output

      ● danted.service - SOCKS (v4 and v5) proxy daemon (danted) Loaded: loaded (/lib/systemd/system/danted.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Wed 2021-12-15 21:48:22 UTC; 1min 45s ago Docs: man:danted(8) man:danted.conf(5) Main PID: 14496 (code=exited, status=1/FAILURE) Dec 15 21:48:21 proxies systemd[1]: Starting SOCKS (v4 and v5) proxy daemon (danted)... Dec 15 21:48:22 proxies systemd[1]: Started SOCKS (v4 and v5) proxy daemon (danted). Dec 15 21:48:22 proxies danted[14496]: Dec 15 21:48:22 (1639604902.102601) danted[14496]: warning: checkconfig(): no socks authentication methods enabled. This means all socks requests will be blocked after negotiation. Perhaps this is not intended?

      To successfully start Dante’s services, you’ll need to enable them in the config file.

      Dante’s config file is provided, by default, in /etc/danted.conf. If you open this file using nano or your favorite text editor, you will see a long list of configuration options, all of them disabled. You could try to navigate through this file and enable some options line-by-line, but in practice it will be more efficient and more readable to delete this file and replace it from scratch. Don’t worry about doing this. You can always review Dante’s default configuration by navigating to its online manual, and you could even redownload the package manually from Ubuntu’s package listing to reobtain the stock configuration file if you ever wanted. In the meantime, go ahead and delete it:

      Now you can replace it with something more concise. Opening a file with a text editor will automatically create the file if it doesn’t exist, so by using nano or your favorite text editor, you should now get an empty configuration file:

      • sudo nano /etc/danted.conf

      Add the following contents:

      /etc/danted.conf

      logoutput: syslog
      user.privileged: root
      user.unprivileged: nobody
      
      # The listening network interface or address.
      internal: 0.0.0.0 port=1080
      
      # The proxying network interface or address.
      external: eth0
      
      # socks-rules determine what is proxied through the external interface.
      socksmethod: username
      
      # client-rules determine who can connect to the internal interface.
      clientmethod: none
      
      client pass {
          from: 0.0.0.0/0 to: 0.0.0.0/0
      }
      
      socks pass {
          from: 0.0.0.0/0 to: 0.0.0.0/0
      }
      

      You now have a usable SOCKS server configuration, running on port 1080, which is a common convention for SOCKS. You can also break down the rest of this configuration file line-by-line:

      • logoutput refers to how Dante will log connections, in this case using regular system logging
      • user.privileged allows dante to have root permissions for checking permissions
      • user.unprivileged does not grant the server any permissions for running as an unprivileged user, as this is unnecessary when not granting more granular permissions
      • internal connection details specify the port that the service is running on and which IP addresses can connect
      • external connection details specify the network interface used for outbound connections, eth0 by default on most servers

      The rest of the configuration details deal with authentication methods, which are discussed in the next section. Don’t forget to open port 1080 in your firewall if you’re using ufw:

      At this point, you could restart Dante and connect to it, but you would have a SOCKS server that’s open to the entire world, which you probably don’t want, so you’ll learn how to secure it first.

      Step 2 — Securing Dante

      If you followed this tutorial so far, Dante will be making use of regular Linux user accounts for authentication. This is helpful, but the password used for that connection will be sent over plain text, so it’s important to create a dedicated SOCKS user that won’t have any other login privileges. To do that, you’ll use useradd with flags that won’t assign a login shell to the user, then set a password:

      • sudo useradd -r -s /bin/false your_dante_user
      • sudo passwd your_dante_user

      You’ll also want to avoid logging into this account over an unsecured wireless connection or sharing the server too widely. Otherwise, malicious actors can and will make repeated efforts to log in.

      Dante supports other authentication methods, but many clients (i.e., applications) that will connect to SOCKS proxies only support basic username and password authentication, so you may want to leave that part as-is. What you can do as an alternative is to restrict access to only specific IP addresses. This isn’t the most sophisticated option, but given the combination of technologies in use here, it’s a sensible one. You may have already learned how to restrict access to specific IP addresses with ufw from our prerequisite tutorials , but you can also do it within Dante directly. Edit your /etc/danted.conf:

      • sudo nano /etc/danted.conf

      /etc/danted.conf

      …
      client pass {
          from: your_ip_address/0 to: 0.0.0.0/0
      }
      

      In order to support multiple IP addresses, you can use CIDR notation, or just add another client pass {} configuration block:

      /etc/danted.conf

      client pass {
          from: your_ip_address/0 to: 0.0.0.0/0
      }
      
      client pass {
          from: another_ip_address/0 to: 0.0.0.0/0
      }
      

      After that, you can finally restart Dante with your configuration changes.

      • sudo systemctl restart danted.service

      This time, when you check the service status, you should see it running without any errors:

      • systemctl status danted.service

      Output

      ● danted.service - SOCKS (v4 and v5) proxy daemon (danted) Loaded: loaded (/lib/systemd/system/danted.service; enabled; vendor preset: enable> Active: active (running) since Thu 2021-12-16 18:06:26 UTC; 24h ago

      In the next step, you’ll connect to your proxy at last.

      Step 3 — Connecting through Dante

      In order to demonstrate your Dante server, you’ll use a command line program called curl, which is popular for making different types of web requests. In general, if you want to verify whether a given connection should be working in a browser under ideal circumstances, you should always test first with curl. You’ll be using curl on your local machine in order to do this – it’s installed by default on all modern Windows, Mac, and Linux environments, so you can open any local shell to run this command:

      • curl -v -x socks5://your_dante_user:your_dante_password@your_server_ip:1080 http://www.google.com/

      Output

      * Trying 138.197.103.77... * TCP_NODELAY set * SOCKS5 communication to www.google.com:80 * SOCKS5 connect to IPv4 142.250.189.228 (locally resolved) * SOCKS5 request granted. * Connected to 138.197.103.77 (138.197.103.77) port 1080 (#0) > GET / HTTP/1.1 …

      The credentials that you used for curl should now work anywhere else you might want to use your new proxy server.

      Conclusion

      In this tutorial, you learned to deploy a popular, open-source API endpoint for proxying traffic with little to no overhead. Many applications have built-in proxy support (often at the OS level) going back decades, making this proxy stack highly reusable.

      Next, you may want to learn how to deploy Squid, an HTTP proxy which can run alongside Dante for proxying different types of web traffic.

      Because one of the most common use cases for proxy servers is proxying traffic to and from different global regions, you may want to review how to use Ansible to automate server deployments next, in case you find yourself wanting to duplicate this configuration in other data centers.



      Source link

      How to Use a Private Go Module in Your Own Project


      The author selected the Diversity in Tech Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      One beneficial aspect of Go’s ecosystem is that a large number of modules are open source. Since they’re open source they can be freely accessed, examined, used, and learned from. However, sometimes it’s necessary to make a private Go module for various reasons, such as keeping proprietary business logic internal to your company.

      In this tutorial, you will publish a private Go module, set up authentication to access a private module, and use a private Go module in a project.

      Prerequisites

      Distributing a Private Module

      Unlike many programming languages, Go distributes modules from repositories instead of a central package server. One benefit of this approach is that publishing a private module is very similar to publishing a public one. Instead of requiring a completely separate private package server, a Go private module is distributed via a private source code repository. Since most source code hosting options support this out of the box, there’s no need to set up an additional private server.

      In order to use a private module, you’ll need to have access to a private Go module. In this section, you’ll create and publish a private module you can use later in the tutorial to access a private module from another Go program.

      To create your new private Go module, start by cloning the private GitHub repository where it will live. As part of the prerequisites you created a private, empty repository named mysecret in your GitHub account and this is the one you will use for your private module. This repository can be cloned anywhere you’d like on your computer, but many developers tend to have a directory for their projects. In this tutorial, you’ll use a directory named projects.

      Make the projects directory and navigate to it:

      • mkdir projects
      • cd projects

      From the projects directory, run git clone to clone your private mysecret repository to your computer:

      • git clone git@github.com:your_github_username/mysecret.git

      Git will confirm it has cloned your module and may warn you that you have cloned an empty repository. If so, this is not something you need to worry about:

      Output

      Cloning into 'mysecret'... warning: You appear to have cloned an empty repository.

      Next, use cd to go into the new mysecret directory you cloned and use go mod init, along with the name of your private repository, to create a new Go module:

      • cd mysecret
      • go mod init github.com/your_github_username/mysecret

      Now that your module is created, it’s time to add a function you can use from another project. Use nano, or your favorite text editor, to open a file with the same name as your repository, such as mysecret.go. The name isn’t significant, and could be anything, but using the same name as the repository makes it easier to determine which file to look in first when working with a new module:

      In the mysecret.go file, name the package with the same name as your repository, then add a SecretProcess function to print the line Running the secret process! when called:

      projects/mysecret/mysecret.go

      package mysecret
      
      import "fmt"
      
      func SecretProcess() {
          fmt.Println("Running the secret process!")
      }
      

      Now that you have your private module created, you will publish it to your private repository for others to use. Since your private repository only allows you to access it initially, you’re able to control who has access to your private module. You might restrict access to yourself, but you could also give access to friends or coworkers as well.

      Since both private and public Go modules are source repositories, publishing a private Go module follows the same process as publishing a public one. To publish your new module, stage your changes in the current directory using the git add command, then commit those changes to your local repository with the git commit command:

      • git add .
      • git commit -m "Initial private module implementation"

      You will see a confirmation from Git that your initial commit has succeeded as well as a summary of the files included in the commit:

      Output

      [main (root-commit) bda059d] Initial private module implementation 2 files changed, 10 insertions(+) create mode 100644 go.mod create mode 100644 mysecret.go

      Now the only part left is to move your changes to your GitHub repository. Similar to a public module, use the git push command to publish your code:

      Git will then push your changes and make them available to anyone with access to your private repository:

      git push origin main
      Enumerating objects: 4, done.
      Counting objects: 100% (4/4), done.
      Delta compression using up to 8 threads
      Compressing objects: 100% (3/3), done.
      Writing objects: 100% (4/4), 404 bytes | 404.00 KiB/s, done.
      Total 4 (delta 0), reused 0 (delta 0), pack-reused 0
      To github.com:your_github_username/mysecret.git
       * [new branch]      main -> main
      

      As with a public Go module, you can also add versions to your private Go module. The Publishing a New Module Version section of the How to Distribute Go Modules tutorial includes information on how to do this.

      In this section, you created a new module with a SecretProcess function and published it to your private mysecret GitHub repository, making it a private Go module. In order to access this module from another Go program, though, you’ll need to configure Go so it knows how to access the module.

      Configuring Go to Access Private Modules

      While Go modules are commonly distributed from their source code repositories, the Go team also runs a few central Go module services to aid ensure modules continue to exist if something happens to the original repository. By default, Go is configured to use these services, but they can cause problems when you try to download a private module because they don’t have access to those modules. To tell Go that some import paths are private and that it shouldn’t try to use the central Go services, you can use the GOPRIVATE environment variable. The GOPRIVATE environment variable is a comma-separated list of import path prefixes where, when encountered, the Go tools will try to access them directly instead of going through the central services. One such example would be the private module you just created.

      In order to use the private module, you will tell Go which path to consider private by setting it in the GOPRIVATE variable. There are a few choices you can make when setting your GOPRIVATE variable values. One option is to set GOPRIVATE to github.com. This might not be what you’re looking for, though, because this would tell Go not to use the central services for any module hosted on github.com, including the ones that aren’t yours.

      The next option would be to set GOPRIVATE to only your own user path, such as github.com/your_github_username. This solves the problem of considering all of GitHub private, but at some point you may have public modules you’ve created that you’d like to download through the Go module mirror. Doing this wouldn’t cause any problems and would be a perfectly reasonable option, but you also have the option of getting even more specific.

      The most specific option would be setting GOPRIVATE to match the path of your module exactly, such as: github.com/your_github_username/mysecret. This solves both of the issues from the previous options, but also introduces the issue that you need to add each of your private repositories to GOPRIVATE individually, such as shown here:

      GOPRIVATE=github.com/your_github_username/mysecret,github.com/your_github_username/othersecret
      

      Choosing the best option for youself is a matter of weighing the pros and cons in your situation.

      Since you only have a single private module now, we’ll use the full repository name for the value. To set the GOPRIVATE=github.com/your_github_username/mysecret environment variable in your current terminal, use the export command:

      • export GOPRIVATE=github.com/your_github_username/mysecret

      If you’d like to double-check that it’s set, you can use the env command along with grep to check for the GOPRIVATE name:

      Output

      GOPRIVATE=github.com/your_github_username/mysecret

      Even though Go now knows your module is private, it’s still not quite enough to use the module yet. If you try to go get your private module into another module, you’ll likely see an error similar to:

      • go get github.com/your_github_username/mysecret

      Output

      go get: module github.com/your_github_username/mysecret: git ls-remote -q origin in /Users/your_github_username/go/pkg/mod/cache/vcs/2f8c...b9ea: exit status 128: fatal: could not read Username for 'https://github.com': terminal prompts disabled Confirm the import path was entered correctly. If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.

      This error message says Go tried to download your module, but it encountered something it still doesn’t have access to. Since Git is being used to download the module, it would usually ask you to enter your credentials. However, in this case, Go is calling Git for you and can’t prompt for them. At this point, to access your module you’ll need to provide a way for Git to retrieve your credentials without your immediate input.

      Providing Private Module Credentials for HTTPS

      One way to tell Git how to log in on your behalf is the .netrc file. Located in a user’s home directory, the .netrc file contains various host names as well as log in credentials for those hosts. It’s widely used by a number of tools, including Git.

      By default, when go get tries to download a module, it will try to use HTTPS first. However, as shown in the previous example, it’s not able to prompt you for your username and password. To give Git your credentials, you’ll need to have a .netrc that includes github.com in your home directory.

      To create a .netrc file on Linux, MacOS, or Windows Subsystem for Linux (WSL), open the .netrc file in your home directory (~/) so you can edit it:

      Next, create a new entry in the file. The machine value should be the hostname you’re setting the credentials for, which is github.com in this case. The login value should then be your GitHub username. Finally, the password value should be the GitHub personal access token you created.

      ~/.netrc

      machine github.com
      login your_github_username
      password your_github_access_token
      

      If you’d prefer, you can also put the entire entry on one line in the file as well:

      ~/.netrc

      machine github.com login your_github_username password your_github_access_token
      

      Note: If you are using Bitbucket for your source code hosting you may also need to add a second entry for api.bitbucket.org in addition to bitbucket.org. In the past, Bitbucket provided hosting for multiple types of version control, so Go would use the API to check the type of repository before trying to download it. While this is no longer the case, the API check still exists. If you encounter this issue, an example error message may look like this:

      go get bitbucket.org/your_github_username/mysecret: reading https://api.bitbucket.org/2.0/repositories/your_bitbucket_username/protocol?fields=scm: 403 Forbidden
          server response: Access denied. You must have write or admin access.
      

      If you see the 403 Forbidden error when trying to download a private module, double check the hostname Go is trying to connect to. It could indicate another hostname, such as api.bitbucket.org, that you need to add to your .netrc file.

      Now your environment is set up to use HTTPS authentication for downloading your private module. Even though HTTPS is the default way Go and Git will try to download a module, it’s also possible to tell Git to use SSH instead. Using SSH instead of HTTPS can be useful so you can use the same SSH key you used to push your private module. It also allows you to use deploy keys when setting up a CI/CD environment if you’d rather not create a personal access token.

      Providing Private Module Credentials for SSH

      To use your SSH key as the authentication method for your private Go module instead of HTTPS, Git provides a configuration option called insteadOf. The insteadOf option allows you to say that “instead of” using https://github.com/ as the request URL for all Git requests, you’d prefer to use ssh://git@github.com/.

      On Linux, MacOS, and WSL this configuration lives in the .gitconfig file. You may already be familiar with this file as it’s also where your commit email address and name are configured as well. To edit the file, use nano, or your favorite text editor, and open the ~/.gitconfig file in your home directory:

      Once you have the file open, edit it to include a url section for ssh://git@github.com/ as in the example below:

      ~/.gitconfig

      [user]
          email = your_github_username@example.com
          name = Sammy the Shark
      
      [url "ssh://git@github.com/"]
          insteadOf = https://github.com/
      

      The order of the url section relative to the user section doesn’t matter, and you also don’t need to worry if there’s nothing else in the file except for the url section you just added. The order of the email and name fields inside the user section also does not matter.

      This new section tells Git that any URL you use that starts with https://github.com/ should have that prefixed replaced with ssh://git@github.com/ instead. Since Go uses HTTPS by default, this also affects your go get commands. Using your private module as an example, this means Go turns the github.com/your_github_username/mysecret import path into the URL https://github.com/your_github_username/mysecret. When Git encounters this URL it will see the URL matches the https://github.com/ prefix referenced by insteadOf and will turn the resulting URL into ssh://git@github.com/your_github_username/mysecret.

      This same pattern can be used for domains other than GitHub as long as the ssh://git@ URL works for that host as well.

      In this section, you configured Git to use SSH to download Go modules by updating your .gitconfig file and adding a url section. Now that authentication for your private module is set up, you can access it for use in your Go programs.

      Using a Private Module

      In the previous sections, you configured Go to access your private Go module via HTTPS, SSH, or possibly both. Now that Go can access your private module, it can be used similar to any public module you may have used in the past. In this section, you’ll create a new Go module that uses your private module.

      In the directory you use for your projects, such as projects, create a directory named myproject for the new project using the mkdir command:

      Once the directory is created, go to the directory using cd and initialize a new Go module using go mod init for your project based on the repository URL your project would live at, such as github.com/your_github_username/myproject. If you don’t plan for your project to be pushed to any other repository the module name could be just myproject, or any other name, but it’s good practice to use full URLs since most modules being shared will need them.

      • cd myproject
      • go mod init github.com/your_github_username/myproject

      Output

      go: creating new go.mod: module github.com/your_github_username/myproject

      Now, create your project’s first code file by opening main.go with nano, or your favorite text editor:

      Inside the file, set up the initial main function you will call your private module from:

      projects/myproject/main.go

      package main
      
      import "fmt"
      
      func main() {
          fmt.Println("My new project!")
      }
      

      To run your project now and make sure everything is set up correctly, you can use the go run command and provide it the main.go file:

      Output

      My new project!

      Next, add your private module as a dependency of your new project using go get, similar to how you would for a public module:

      • go get github.com/your_github_username/mysecret

      The go tool will then download your private module’s code and add it as a dependency using a version string matching your latest commit hash and the time of that commit:

      Output

      go: downloading github.com/your_github_username/mysecret v0.0.0-20210920195630-bda059d63fa2 go get: added github.com/your_github_username/mysecret v0.0.0-20210920195630-bda059d63fa2

      Finally, open the main.go file again and update it to add a call to your private module’s SecretProcess function in the main function. You’ll also need to update the import statement to add your github.com/your_github_username/mysecret private module as an import as well:

      projects/myproject/main.go

      package main
      
      import (
          "fmt"
      
          "github.com/your_github_username/mysecret"
      )
      
      func main() {
          fmt.Println("My new project!")
          mysecret.SecretProcess()
      }
      

      To see the final project running with your private module, use the go run command again while providing the main.go file as the parameter:

      You will see the My new project! line from the original code, but now you’ll also see a Running the secret process! line from your imported mysecret module as well:

      Output

      My new project! Running the secret process!

      In this section, you used go init to create a new Go module to access the private module you published earlier. Once you had the module created, you then used go get to download your private module as you would with a public Go module. Finally, you used go run to compile and run your Go program using the private module.

      Conclusion

      In this tutorial, you created and published a private Go module. You also set up both HTTPS and SSH authentication to access your private Go module. Finally, you used your private module in a new project.

      For more information on Go modules, the Go project has a series of blog posts detailing how the Go tools interact with and understand modules. The Go project also has a very detailed and technical reference for Go modules in the Go Modules Reference.

      In addition to the GOPRIVATE environment variable, more variables are available to use when working with private Go modules. They can be seen in detail in the Private Modules section of the Go Modules Reference.

      If you’re interested in exploring the .netrc file in more detail, the GNU website on .netrc includes a list of all the available keywords. The git-config documentation also includes more information about how the insteadOf configuration option you used works in addition to other options that are available.

      This tutorial is also part of the DigitalOcean How to Code in Go series. The series covers a number of Go topics, from installing Go for the first time to how to use the language itself.



      Source link