One place for hosting & domains

      Download

      How To Use Wget to Download Files and Interact with REST APIs


      The author selected the COVID-19 Relief Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Wget is a networking command-line tool that lets you download files and interact with REST APIs. It supports the HTTP,HTTPS, FTP, and FTPS internet protocols. Wget can deal with unstable and slow network connections. In the event of a download failure, Wget keeps trying until the entire file has been retrieved. Wget also lets you resume a file download that was interrupted without starting from scratch.

      You can also use Wget to interact with REST APIs without having to install any additional external programs. You can make GET, POST, PUT, and DELETE HTTP requests with single and multiple headers right in the terminal.

      In this tutorial, you will use Wget to download files, interact with REST API endpoints, and create and manage a Droplet in your DigitalOcean account.

      To follow along with this tutorial using a terminal in your browser, click the Launch an Interactive Terminal! button below:

      Launch an Interactive Terminal!

      Otherwise, if you’d like to use your local system or a remote server, open a terminal and run the commands there.

      Prerequisites

      To complete this tutorial, you will need:

      • Wget installed. Most Linux distributions have Wget installed by default. To check, type wget in your terminal and press ENTER. If it is not installed, it will display: command not found. You can install it by running the following command: sudo apt-get install wget.

      • A DigitalOcean account. If you do not have one, sign up for a new account.

      • A DigitalOcean Personal Access Token, which you can create via the DigitalOcean control panel. Instructions to do that can be found here: How to Generate a Personal Access Token.

      Downloading Files

      In this section, you will use Wget to customize your download experience. For example, you will learn to download a single file and multiple files, handle file downloads in unstable network conditions, and, in the case of a download interruption, resume a download.

      First, create a directory to save the files that you will download throughout this tutorial:

      • mkdir -p DigitalOcean-Wget-Tutorial/Downloads

      With the command above, you have created a directory named DigitalOcean-Wget-Tutorial, and inside of it, you created a subdirectory named Downloads. This directory and its subdirectory will be where you will store the files you download.

      Navigate to the DigitalOcean-Wget-Tutorial directory:

      • cd DigitalOcean-Wget-Tutorial

      You have successfully created the directory where you will store the files you download.

      Downloading a file

      In order to download a file using Wget, type wget followed by the URL of the file that you wish to download. Wget will download the file in the given URL and save it in the current directory.

      Let’s download a minified version of jQuery using the following command:

      • wget https://code.jquery.com/jquery-3.6.0.min.js

      Don’t worry if you don’t know what jQuery is – you could have downloaded any file available on the internet. All you need to know is that you successfully used Wget to download a file from the internet.

      The output will look similar to this:

      Output

      --2021-07-21 16:25:11-- https://code.jquery.com/jquery-3.6.0.min.js Resolving code.jquery.com (code.jquery.com)... 69.16.175.10, 69.16.175.42, 2001:4de0:ac18::1:a:1a, ... Connecting to code.jquery.com (code.jquery.com)|69.16.175.10|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 89501 (87K) [application/javascript] Saving to: ‘jquery-3.6.0.min.js’ jquery-3.6.0.min.js 100%[===================>] 87.40K 114KB/s in 0.8s 2021-07-21 16:25:13 (114 KB/s) - ‘jquery-3.6.0.min.js’ saved [89501/89501]

      According to the output above, you have successfully downloaded and saved a file named jquery-3.6.0.min.js to your current directory.

      You can check the contents of the current directory using the following command:

      The output will look similar to this:

      Output

      Downloads jquery-3.6.0.min.js

      Specifying the filename for the downloaded file

      When downloading a file, Wget defaults to storing it using the name that the file has on the server. You can change that by using the -O option to specify a new name.

      Download the jQuery file you downloaded previously, but this time save it under a different name:

      • wget -O jquery.min.js https://code.jquery.com/jquery-3.6.0.min.js

      With the command above, you set the jQuery file to be saved as jquery.min.js instead of jquery-3.6.0.min.js

      The output will look similar to this:

      Output

      --2021-07-21 16:27:01-- https://code.jquery.com/jquery-3.6.0.min.js Resolving code.jquery.com (code.jquery.com)... 69.16.175.10, 69.16.175.42, 2001:4de0:ac18::1:a:2b, ... Connecting to code.jquery.com (code.jquery.com)|69.16.175.10|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 89501 (87K) [application/javascript] Saving to: ‘jquery.min.js’ jquery.min.js 100%[==================================>] 87.40K 194KB/s in 0.4s 2021-07-21 16:27:03 (194 KB/s) - ‘jquery.min.js’ saved [89501/89501]

      According to the output above, you have successfully downloaded the jQuery file and saved it as jquery.min.js.

      You can use the ls command to list the contents of your current directory, and you will see the jquery.min.js file there:

      The output will look similar to this:

      Output

      Downloads jquery-3.6.0.min.js jquery.min.js

      So far, you have used wget to download files to the current directory. Next, you will download to a specific directory.

      Downloading a file to a specific directory

      When downloading a file, Wget stores it in the current directory by default. You can change that by using the -P option to specify the name of the directory where you want to save the file.

      Download the jQuery file you downloaded previously, but this time save it in the Downloads subdirectory.

      • wget -P Downloads/ https://code.jquery.com/jquery-3.6.0.min.js

      The output will look similar to this:

      Output

      --2021-07-21 16:28:50-- https://code.jquery.com/jquery-3.6.0.min.js Resolving code.jquery.com (code.jquery.com)... 69.16.175.42, 69.16.175.10, 2001:4de0:ac18::1:a:2b, ... Connecting to code.jquery.com (code.jquery.com)|69.16.175.42|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 89501 (87K) [application/javascript] Saving to: ‘Downloads/jquery-3.6.0.min.js’ jquery-3.6.0.min.js 100%[==================================>] 87.40K 43.6KB/s in 2.0s 2021-07-21 16:28:53 (43.6 KB/s) - ‘Downloads/jquery-3.6.0.min.js’ saved [89501/89501]

      Notice the last line where it says that the jquery-3.6.0.min.js file was saved in the Downloads directory.

      If you use the ls Downloads command to list the contents of the Downloads directory, you will see the jQuery file there:

      Run the ls command:

      The output will look similar to this:

      Output

      jquery-3.6.0.min.js

      Turning Wget’s output off

      By default, Wget outputs a lot of information to the terminal when you download a file. You can use the -q option to turn off all output.

      Download the jQuery file, but this time without showing any output:

      • wget -q https://code.jquery.com/jquery-3.6.0.min.js

      You won’t see any output, but if you use the ls command to list the contents of the current directory you will find a file named jquery-3.6.0.min.js.1:

      The output will look similar to this:

      Output

      Downloads jquery-3.6.0.min.js jquery-3.6.0.min.js.1 jquery.min.js

      Before saving a file, Wget checks whether the file exists in the desired directory. If it does, Wget adds a number to the end of the file. If you ran the command above one more time, Wget would create a file named jquery-3.6.0.min.js.2. This number increases every time you download a file to a directory that already has a file with the same name.

      You have successfully turned off Wget’s output, but now you can’t monitor the download progress. Let’s look at how to show the download progress bar.

      Showing the download progress bar

      Wget lets you show the download progress bar but hide any other output by using the -q option alongside the --show-progress option.

      Download the jQuery file, but this time only show the download progress bar:

      • wget -q --show-progress https://code.jquery.com/jquery-3.6.0.min.js

      The output will look similar to this:

      Output

      jquery-3.6.0.min.js.2 100%[================================================>] 87.40K 207KB/s in 0.4s

      Use the ls command to check the contents of the current directory and you will find the file you have just downloaded with the name jquery-3.6.0.min.js.2

      From this point forward you will be using the -q and --show-progress options in most of the subsequent Wget commands.

      So far you have only downloaded a single file. Next, you will download multiple files.

      Downloading multiple files

      In order to download multiples files using Wget, you need to create a .txt file and insert the URLs of the files you wish to download. After inserting the URLs inside the file, use the wget command with the -i option followed by the name of the .txt file containing the URLs.

      Create a file named images.txt:

      In images.txt, add the following URLs:

      images.txt

      https://cdn.pixabay.com/photo/2016/12/13/05/15/puppy-1903313__340.jpg
      https://cdn.pixabay.com/photo/2016/01/05/17/51/maltese-1123016__340.jpg
      https://cdn.pixabay.com/photo/2020/06/30/22/34/dog-5357794__340.jpg
      

      The URLs link to three random images of dogs found on Pixabay. After you have added the URLs, save and close the file.

      Now you will use the -i option alongside the -P,-q and --show-progress options that you learned earlier to download all three images to the Downloads directory:

      • wget -i images.txt -P Downloads/ -q --show-progress

      The output will look similar to this:

      Output

      puppy-1903313__340.jp 100%[=========================>] 26.44K 93.0KB/s in 0.3s maltese-1123016__340. 100%[=========================>] 50.81K --.-KB/s in 0.06s dog-5357794__340.jpg 100%[=========================>] 30.59K --.-KB/s in 0.07s

      If you use the ls Downloads command to list the contents of the Downloads directory, you will find the names of the three images you have just downloaded:

      The output will look similar to this:

      Output

      dog-5357794__340.jpg jquery-3.6.0.min.js maltese-1123016__340.jpg puppy-1903313__340.jpg

      Limiting download speed

      So far, you have download files with the maximum available download speed. However, you might want to limit the download speed to preserve resources for other tasks. You can limit the download speed by using the --limit-rate option followed by the maximum speed allowed in kiloBits per second and the letter k.

      Download the first image in the images.txt file with a speed of 15 kB/S to the Downloads directory:

      • wget --limit-rate 15k -P Downloads/ -q --show-progress https://cdn.pixabay.com/photo/2016/12/13/05/15/puppy-1903313__340.jpg

      The output will look similar to this:

      Output

      puppy-1903313__340.jpg.1 100%[====================================================>] 26.44K 16.1KB/s in 1.6s

      If you use the ls Downloads command to check the contents of the Downloads directory, you will see the file you have just downloaded with the name puppy-1903313__340.jpg.1.

      When downloading a file that already exists, Wget creates a new file instead of overwriting the existing file. Next, you will overwrite a downloaded file.

      Overwriting a downloaded file

      You can overwrite a file you have downloaded by using the -O option alongside the name of the file. In the code below, you will first download the second image listed in the images.txt file to the current directory and then you will overwrite it.

      First, download the second image to the current directory and set the name to image2.jpg:

      • wget -O image2.jpg -q --show-progress https://cdn.pixabay.com/photo/2016/12/13/05/15/puppy-1903313__340.jpg

      The output will look similar to this::

      Output

      image2.jpg 100%[====================================================>] 26.44K --.-KB/s in 0.04s

      If you use the ls command to check the contents of the current directory, you will see the file you have just downloaded with the name image2.jpg.

      If you wish to overwrite this image2.jpg file, you can run the same command you ran earlier :

      • wget -O image2.jpg -q --show-progress https://cdn.pixabay.com/photo/2016/12/13/05/15/puppy-1903313__340.jpg

      You can run the command above as many times as you like and Wget will download the file and overwrite the existing one. If you run the command above without the -O option, Wget will create a new file each time you run it.

      Resuming a download

      Thus far, you have successfully downloaded multiple files without interruption. However, if the download was interrupted, you can resume it by using the -c option.

      Run the following command to download a random image of a dog found on Pixabay. Note that in the command, you have set the maximum speed to 1 KB/S. Before the image finishes downloading, press Ctrl+C to cancel the download:

      • wget --limit-rate 1k -q --show-progress https://cdn.pixabay.com/photo/2018/03/07/19/51/grass-3206938__340.jpg

      To resume the download, pass the -c option. Note that this will only work if you run this command in the same directory as the incomplete file:

      • wget -c --limit-rate 1k -q --show-progress https://cdn.pixabay.com/photo/2018/03/07/19/51/grass-3206938__340.jpg

      Up until now, you have only downloaded files in the foreground. Next, you will download files in the background.

      Downloading in the background

      You can download files in the background by using the -b option.

      Run the command below to download a random image of a dog from Pixabay in the background:

      • wget -b https://cdn.pixabay.com/photo/2018/03/07/19/51/grass-3206938__340.jpg

      When you download files in the background, Wget creates a file named wget-log in the current directory and redirects all output to this file. If you wish to watch the status of the download, you can use the following command:

      The output will look similar to this:

      Output

      Resolving cdn.pixabay.com (cdn.pixabay.com)... 104.18.20.183, 104.18.21.183, 2606:4700::6812:14b7, ... Connecting to cdn.pixabay.com (cdn.pixabay.com)|104.18.20.183|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 33520 (33K) [image/jpeg] Saving to: ‘grass-3206938__340.jpg’ 0K .......... .......... .......... .. 100% 338K=0.1s 2021-07-20 23:49:52 (338 KB/s) - ‘grass-3206938__340.jpg’ saved [33520/33520]

      Setting a timeout

      Until this point, we have assumed that the server that you are trying to download files from is working properly. However, let’s assume that the server is not working properly. You can use Wget to first limit the amount of time that you wait for the server to respond and then limit the number of times that Wget tries to reach the server.

      If you wish to download a file but you are unsure if the server is working properly, you can set a timeout by using the -T option followed by the time in seconds.

      In the following command, you are setting the timeout to 5 seconds:

      • wget -T 5 -q --show-progress https://cdn.pixabay.com/photo/2016/12/13/05/15/puppy-1903313__340.jpg

      Setting maximum number of tries

      You can also set how many times Wget attempts to download a file after being interrupted by passing the --tries option followed by the number of tries.

      By running the command below, you are limiting the number of tries to 3:

      • wget --tries=3 -q --show-progress https://cdn.pixabay.com/photo/2018/03/07/19/51/grass-3206938__340.jpg

      If you would like to try indefinitely you can pass inf alongside the --tries option:

      • wget --tries=inf -q --show-progress https://cdn.pixabay.com/photo/2018/03/07/19/51/grass-3206938__340.jpg

      In this section, you used Wget to download a single file and multiple files, resume downloads, and handle network issues. In the next section, you will learn to interact with REST API endpoints.

      Interacting with REST APIs

      In this section, you will use Wget to interact with REST APIs without having to install an external program. You will learn the syntax to send the most commonly used HTTP methods: GET, POST, PUT, and DELETE.

      We are going to use JSONPlaceholder as the mock REST API. JSONPlaceholder is a free online REST API that you can use for fake data. (The requests you send to it won’t affect any databases and the data won’t be saved.)

      Sending GET requests

      Wget lets you send GET requests by running a command that looks like the following:

      In the command above, the - after the -O option means standard output, so Wget will send the output of the URL to the terminal instead of sending it to a file as you did in the previous section. GET is the default HTTP method that Wget uses.

      Run the following command in the terminal window:

      • wget -O- https://jsonplaceholder.typicode.com/posts?_limit=2

      In the command above, you used wget to send a GET request to JSON Placeholder in order to retrieve two posts from the REST API.

      The output will look similar to this:

      Output

      --2021-07-21 16:52:51-- https://jsonplaceholder.typicode.com/posts?_limit=2 Resolving jsonplaceholder.typicode.com (jsonplaceholder.typicode.com)... 104.21.10.8, 172.67.189.217, 2606:4700:3032::6815:a08, ... Connecting to jsonplaceholder.typicode.com (jsonplaceholder.typicode.com)|104.21.10.8|:443... connected. HTTP request sent, awaiting response... 200 OK' Length: 600 [application/json] Saving to: ‘STDOUT’ - 0%[ ] 0 --.-KB/s [ { "userId": 1, "id": 1, "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit", "body": "quia et suscipitnsuscipit recusandae consequuntur expedita et cumnreprehenderit molestiae ut ut quas totamnnostrum rerum est autem sunt rem eveniet architecto" }, { "userId": 1, "id": 2, "title": "qui est esse", "body": "est rerum tempore vitaensequi sint nihil reprehenderit dolor beatae ea dolores nequenfugiat blanditiis voluptate porro vel nihil molestiae ut reiciendisnqui aperiam non debitis possimus qui neque nisi nulla" } - 100%[==================================>] 600 --.-KB/s in 0s 2021-07-21 16:52:53 (4.12 MB/s) - written to stdout [600/600]

      Notice the line where it says HTTP request sent, awaiting response... 200 OK, which means that you have successfully sent a GET request to JSONPlaceholder.

      If that is too much output you can use the -q option that you learned in the previous section to restrict the output to the results of the GET request:

      • wget -O- -q https://jsonplaceholder.typicode.com/posts?_limit=2

      The output will look similar to this:

      Output

      [ { "userId": 1, "id": 1, "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit", "body": "quia et suscipitnsuscipit recusandae consequuntur expedita et cumnreprehenderit molestiae ut ut quas totamnnostrum rerum est autem sunt rem eveniet architecto" }, { "userId": 1, "id": 2, "title": "qui est esse", "body": "est rerum tempore vitaensequi sint nihil reprehenderit dolor beatae ea dolores nequenfugiat blanditiis voluptate porro vel nihil molestiae ut reiciendisnqui aperiam non debitis possimus qui neque nisi nulla" } ]

      Sending POST requests

      Wget lets you send POST requests by running a command that looks like the following:

      • wget --method==[post] -O- --body-data=[ body in json format ] --header=[ String ] [ URL ]

      Run the following command:

      • wget --method=post -O- -q --body-data="{"title": "Wget POST","body": "Wget POST example body","userId":1}" --header=Content-Type:application/json https://jsonplaceholder.typicode.com/posts

      In the command above, you used wget to send a POST request to JSON Placeholder to create a new post. You set the method to post, the Header to Content-Type:application/json and sent the following request body to it :{"title": "Wget POST","body": "Wget POST example body","userId":1}.

      The output will look similar to this:

      Output

      { "title": "Wget POST", "body": "Wget POST example body", "userId": 1, "id": 101 }

      Sending PUT requests

      Wget lets you send PUT requests by running a command that looks like the following:

      • wget --method==[put] -O- --body-data=[ body in json format ] --header=[ String ] [ URL ]

      Run the following command:

      • wget --method=put -O- -q --body-data="{"title": "Wget PUT", "body": "Wget PUT example body", "userId": 1, "id":1}" --header=Content-Type:application/json https://jsonplaceholder.typicode.com/posts/1

      In the command above you used wget to send a PUT request to JSON Placeholder to edit the first post in this REST API. You set the method to put, the Header to Content-Type:application/json and sent the following request body to it :{"title": "Wget PUT", "body": "Wget PUT example body", "userId": 1, "id":1} .

      The output will look similar to this:

      Output

      { "body": "Wget PUT example body", "title": "Wget PUT", "userId": 1, "id": 1 }

      Sending DELETE requests

      Wget lets you send DELETE requests by running a command that looks like the following:

      • wget --method==[delete] -O- [ URL ]

      Run the following command:

      • wget --method=delete -O- -q --header=Content-Type:application/json https://jsonplaceholder.typicode.com/posts/1

      In the command above you used wget to send a DELETE request to JSON Placeholder to delete the first post in this REST API. You set the method to delete, and set the post you want to delete to 1 in the URL.

      The output will look similar to this:

      Output

      {}

      In this section, you learned how to use Wget to send GET, POST, PUT and DELETE requests with only one header field. In the next section, you will learn how to send multiple header fields in order to create and manage a Droplet in your DigitalOcean account.

      Creating and Managing a DigitalOcean Droplet

      In this section, you will apply what you learned in the previous section and use Wget to create and manage a Droplet in your DigitalOcean account. But before you do that, you will learn how to send multiple headers fields in a HTTP method.

      The syntax for a command to send multiple headers looks like this:

      • wget --header=[ first header ] --header=[ second header] --header=[ N header] [ URL ]

      You can have as many headers fields as you like by repeating the --header option as many times as you need.

      To create a Droplet or interact with any other resource in the DigitalOcean API, you will need to send two request headers:

      Content-Type: application/json
      Authorization: Bearer your_personal_access_token
      

      You already saw the first header in the previous section. The second header is what lets you authenticate your account. It has the String named Bearer followed by your DigitalOcean account Personal Access Token.

      Run the following command, replacing your_personal_access_token with your DigitalOcean Personal Access Token:

      • wget --method=post -O- -q --header="Content-Type: application/json" --header="Authorization: Bearer your_personal_access_token" --body-data="{"name":"Wget-example","region":"nyc1","size":"s-1vcpu-1gb","image":"ubuntu-20-04-x64","tags": ["Wget-tutorial"]}" https://api.digitalocean.com/v2/droplets

      With the command above, you have created an ubuntu-20-04-x64 Droplet in the nyc1 region named Wget-example with 1vcpu and 1gb of memory, and you have set the tag to Wget-tutorial. For more information about the attributes in the body-data field, see the DigitalOcean API documentation.

      The output will look similar to this:

      Output

      {"droplet":{"id":237171073,"name":"Wget-example","memory":1024,"vcpus":1,"disk":25,"locked":false,"status":"new","kernel":null,"created_at":"2021-03-16T12:38:59Z","features":[],"backup_ids":[],"next_backup_window":null,"snapshot_ids":[],"image":{"id":72067660,"name":"20.04 (LTS) x64","distribution":"Ubuntu","slug":"ubuntu-20-04-x64","public":true,"regions":["nyc3","nyc1","sfo1","nyc2","ams2","sgp1","lon1","ams3","fra1","tor1","sfo2","blr1","sfo3"],"created_at":"2020-10-20T16:34:30Z","min_disk_size":15,"type":"base","size_gigabytes":0.52,"description":"Ubuntu 20.04 x86","tags":[],"status":"available"},"volume_ids":[],"size":{"slug":"s-1vcpu-1gb","memory":1024,"vcpus":1,"disk":25,"transfer":1.0,"price_monthly":5.0,"price_hourly":0.00744,"regions":["ams2","ams3","blr1","fra1","lon1","nyc1","nyc2","nyc3","sfo1","sfo3","sgp1","tor1"],"available":true,"description":"Basic"},"size_slug":"s-1vcpu-1gb","networks":{"v4":[],"v6":[]},"region":{"name":"New York 1","slug":"nyc1","features":["backups","ipv6","metadata","install_agent","storage","image_transfer"],"available":true,"sizes":["s-1vcpu-1gb","s-1vcpu-1gb-intel","s-1vcpu-2gb","s-1vcpu-2gb-intel","s-2vcpu-2gb","s-2vcpu-2gb-intel","s-2vcpu-4gb","s-2vcpu-4gb-intel","s-4vcpu-8gb","c-2","c2-2vcpu-4gb","s-4vcpu-8gb-intel","g-2vcpu-8gb","gd-2vcpu-8gb","s-8vcpu-16gb","m-2vcpu-16gb","c-4","c2-4vcpu-8gb","s-8vcpu-16gb-intel","m3-2vcpu-16gb","g-4vcpu-16gb","so-2vcpu-16gb","m6-2vcpu-16gb","gd-4vcpu-16gb","so1_5-2vcpu-16gb","m-4vcpu-32gb","c-8","c2-8vcpu-16gb","m3-4vcpu-32gb","g-8vcpu-32gb","so-4vcpu-32gb","m6-4vcpu-32gb","gd-8vcpu-32gb","so1_5-4vcpu-32gb","m-8vcpu-64gb","c-16","c2-16vcpu-32gb","m3-8vcpu-64gb","g-16vcpu-64gb","so-8vcpu-64gb","m6-8vcpu-64gb","gd-16vcpu-64gb","so1_5-8vcpu-64gb","m-16vcpu-128gb","c-32","c2-32vcpu-64gb","m3-16vcpu-128gb","m-24vcpu-192gb","g-32vcpu-128gb","so-16vcpu-128gb","m6-16vcpu-128gb","gd-32vcpu-128gb","m3-24vcpu-192gb","g-40vcpu-160gb","so1_5-16vcpu-128gb","m-32vcpu-256gb","gd-40vcpu-160gb","so-24vcpu-192gb","m6-24vcpu-192gb","m3-32vcpu-256gb","so1_5-24vcpu-192gb"]},"tags":["Wget-tutorial"]},"links":{"actions":[{"id":1164336542,"rel":"create","href":"https://api.digitalocean.com/v2/actions/1164336542"}]}}

      If you see an output similar to the one above that means that you have successfully created a Droplet.

      Now let’s get a list of all the Droplets in your account that have the tag Wget-tutorial. Run the following command, replacing your_personal_access_token with your DigitalOcean Personal Access Token:

      • wget -O- -q --header="Content-Type: application/json" --header="Authorization: Bearer your_personal_access_token" https://api.digitalocean.com/v2/droplets?tag_name=Wget-tutorial

      You should see the name of the Droplet you have just created in the output:

      Output

      {"droplets":[{"id":237171073,"name":"Wget-example","memory":1024,"vcpus":1,"disk":25,"locked":false,"status":"active","kernel":null,"created_at":"2021-03-16T12:38:59Z","features":["private_networking"],"backup_ids":[],"next_backup_window":null,"snapshot_ids":[],"image":{"id":72067660,"name":"20.04 (LTS) x64","distribution":"Ubuntu","slug":"ubuntu-20-04-x64","public":true,"regions":["nyc3","nyc1","sfo1","nyc2","ams2","sgp1","lon1","ams3","fra1","tor1","sfo2","blr1","sfo3"],"created_at":"2020-10-20T16:34:30Z","min_disk_size":15,"type":"base","size_gigabytes":0.52,"description":"Ubuntu 20.04 x86","tags":[],"status":"available"},"volume_ids":[],"size":{"slug":"s-1vcpu-1gb","memory":1024,"vcpus":1,"disk":25,"transfer":1.0,"price_monthly":5.0,"price_hourly":0.00744,"regions":["ams2","ams3","blr1","fra1","lon1","nyc1","nyc2","nyc3","sfo1","sfo3","sgp1","tor1"],"available":true,"description":"Basic"},"size_slug":"s-1vcpu-1gb","networks":{"v4":[{"ip_address":"10.116.0.2","netmask":"255.255.240.0","gateway":"","type":"private"},{"ip_address":"204.48.20.197","netmask":"255.255.240.0","gateway":"204.48.16.1","type":"public"}],"v6":[]},"region":{"name":"New York 1","slug":"nyc1","features":["backups","ipv6","metadata","install_agent","storage","image_transfer"],"available":true,"sizes":["s-1vcpu-1gb","s-1vcpu-1gb-intel","s-1vcpu-2gb","s-1vcpu-2gb-intel","s-2vcpu-2gb","s-2vcpu-2gb-intel","s-2vcpu-4gb","s-2vcpu-4gb-intel","s-4vcpu-8gb","c-2","c2-2vcpu-4gb","s-4vcpu-8gb-intel","g-2vcpu-8gb","gd-2vcpu-8gb","s-8vcpu-16gb","m-2vcpu-16gb","c-4","c2-4vcpu-8gb","s-8vcpu-16gb-intel","m3-2vcpu-16gb","g-4vcpu-16gb","so-2vcpu-16gb","m6-2vcpu-16gb","gd-4vcpu-16gb","so1_5-2vcpu-16gb","m-4vcpu-32gb","c-8","c2-8vcpu-16gb","m3-4vcpu-32gb","g-8vcpu-32gb","so-4vcpu-32gb","m6-4vcpu-32gb","gd-8vcpu-32gb","so1_5-4vcpu-32gb","m-8vcpu-64gb","c-16","c2-16vcpu-32gb","m3-8vcpu-64gb","g-16vcpu-64gb","so-8vcpu-64gb","m6-8vcpu-64gb","gd-16vcpu-64gb","so1_5-8vcpu-64gb","m-16vcpu-128gb","c-32","c2-32vcpu-64gb","m3-16vcpu-128gb","m-24vcpu-192gb","g-32vcpu-128gb","so-16vcpu-128gb","m6-16vcpu-128gb","gd-32vcpu-128gb","m3-24vcpu-192gb","g-40vcpu-160gb","so1_5-16vcpu-128gb","m-32vcpu-256gb","gd-40vcpu-160gb","so-24vcpu-192gb","m6-24vcpu-192gb","m3-32vcpu-256gb","so1_5-24vcpu-192gb"]},"tags":["Wget-tutorial"],"vpc_uuid":"5ee0a168-39d1-4c60-a89c-0b47390f3f7e"}],"links":{},"meta":{"total":1}}

      Now let’s take the id of the Droplet you have created and use it to delete the Droplet. Run the following command, replacing your_personal_access_token with your DigitalOcean Personal Access Token and your_droplet_id with your Droplet id:

      • wget --method=delete -O- --header="Content-Type: application/json" --header="Authorization: Bearer your_personal_access_token" https://api.digitalocean.com/v2/droplets/your_droplet_id

      In the command above, you added your Droplet id to the URL to delete it. If you are seeing a 204 No Content in the output, that means that you succeeded in deleting the Droplet.

      In this section, you used Wget to send multiple headers. Then, you created and managed a Droplet in your DigitalOcean account.

      Conclusion

      In this tutorial, you used Wget to download files in stable and unstable network conditions and interact with REST API endpoints. You then used this knowledge to create and manage a Droplet in your DigitalOcean account. If you would like to learn more about Wget, visit this tool’s manual page. For more Linux command-line tutorials visit DigitalOcean community tutorials.



      Source link

      Download a Local Copy of your Linode Backup


      Updated by Linode

      Written by Edward Angert


      Use promo code DOCS10 for $10 credit on a new account.

      The Linode Backups service can create automatic and manual snapshots of your Linode. A completed backup can be directly restored to the origin Linode or to a new Linode in the same data center. These workflows make it easy to revert to a working configuration if you run into any unexpected issues with your software.

      Linode’s backups are stored in a way that is only directly readable by the Linode Backups service. A common question for the service is how you can download the content from your Linode Backups to another storage location, like your home computer. This can be accomplished in two phases:

      1. Restore a backup to a new Linode in the same data center.
      2. Download either specific files or the entire disk image from that Linode, as needed.

      Before You Begin

      Account Permissions and Billing

      Several of the steps in this guide involve adding services to or removing services from a Linode account. Visit our guide on Users and Permissions for more information about restricted Linode users.

      Note that the cost of adding Backups service and adding a Linode to your account is billed, prorated per hour. If the backups service is only enabled for a few hours, you will only be charged for a few hours of the service. See the Backups pricing details for more information. Likewise, when you create a Linode, you will be billed per hour that the Linode exists, whether it is powered on or not.

      The steps in this guide have been designed to minimize the potential costs associated with this process. Additionally, keep the following in mind:

      • Removing a Linode from your account also cancels the associated Backup service for that Linode.
      • A Linode’s backups are deleted when a Linode is deleted.
      • If you choose to leave the Backups service enabled, or if you do not remove the additional Linode from your account, you will be automatically billed. If you only power the Linode off, you will still be billed for it.

      Enable Backups and Take a Snapshot

      These steps are the minimum required for the scope of this guide. Visit our Backups guide for information about how to implement regular backups of your Linode.

      1. Go to your Linode’s dashboard, click Backups, click Enable backups for this Linode » and confirm the additional cost per month.

      2. This guide focuses on saving a snapshot or specific backup. Click Take a New Snapshot Now.

        • The snapshot appears in the Backup History at the bottom of the page.

      Restore from a Backup

      This section shows how to restore a backup to a new Linode, or to an existing Linode.

      Restoring a backup will create a new configuration profile and a new set of disks on your Linode. The restore process does not restore single files or directories automatically. Restoring particular files can be done by completing a normal restore, copying the files off of the new disks, and then removing the disks afterward.

      Note

      The size of the disk(s) created by the restore process will only be slightly larger than the total size of the files restored. This means that the disk(s) created will be ‘full’.

      Some applications, like databases, need some amount of free unused space inside the disk in order to run. As a result, you may want to increase your disk(s) size after the restore process is completed.

      To restore a backup to a different data center, first restore to a Linode in the same data center, creating a new one if necessary. Once the restore is complete, use the Clone tab to copy the disk(s) to a Linode in a different data center.

      Restore to a New Linode

      You can restore a backup to any Linode located in the same data center, even if the target does not have the Backup Service enabled. This section covers how to restore a backup to a new Linode that does not have any disks deployed to it. If you wish to restore your backup to an existing Linode, see the next section.

      1. From the Dashboard of the Linode whose backups you intend to restore, click on the Backups tab. Select the Restore to… link beneath the backup version that you want to restore.

      2. Under the Select column, click the Restore to this Linode link next to your new Linode:

        The backup disks and configuration profiles will be restored to the Linode you selected. Watch the Host Job Queue to monitor the progress. Restoring from a backup can take several minutes depending on the size of your Linode and the amount of data you have stored on it.

      Restore to an Existing Linode

      To restore a backup to an existing Linode, you will need to make sure that you have enough storage space that is not currently assigned to disk images.

      Note

      If you are attempting to restore a disk to the same Linode the backup was created from, the restoration process will not delete the original disk for you. Manually delete the original disk to make room for the backup.

      1. Start by confirming the size of the backup that you wish to restore. From the Backups tab in your Linode’s Dashboard, click the Restore to… link beneath your desired backup version.

      2. Check the Total size required field to confirm the size of your backup.

        As an example, if the total size of the backup comes to 3107MB, this means you would need at least that much free space to restore the backup to your Linode.

      3. Next, you’ll confirm the total space assigned to disk images on your Linode, via the Storage indicator on your Linode’s Dashboard.

      4. If the amount of space available is greater than the size of the backup, you can proceed with restoring. If the amount of unallocated space is less than the size of the backup, you can resize your existing disks to make room for it.

        Note

        In some cases, you will not be able to shrink your disks enough to fit the restored backup. As an alternative, you can change your Linode’s plan to a higher tier that offers more disk space.
      5. Once the disk resize has completed, check the storage indicator on your Linode’s Dashboard to confirm that you’ve freed up enough space for your backup.

      6. From the Backups tab, click the Restore to this Linode link next to your Linode.

        Your backup will begin restoring to your Linode, and you can monitor its progress from the Host Job Queue in your Linode’s Dashboard tab. Note that the time it takes to restore your backup will vary depending upon the restore size, and the number of files being restored.

      Download Specific Files or Directories over SSH

      If you just need specific files from your Linode, you can download those over SSH. In order to do so, you’ll first need to reboot your Linode under the new configuration profile that was created by the restore process. This new profile is assigned to the restored disks, and your backed up data will be accessible when you boot from them.

      Downloading files over SSH can be done at a command-line interface, or with a graphical SFTP file browser.

      Using SCP

      To retrieve a specific directory or file via the command-line, you can use the secure copy (SCP) command from your computer. SCP is installed by default on most Mac and Linux systems, and you can install a tool like Cygwin to use it on Windows.

      • The syntax for using SCP to copy a file from your Linode into a directory on your computer is:

        scp your_linode_username@your_linode_ip:/path/to/your/file.txt /path/to/your/local/directory/
        

        The file will be saved inside /path/to/your/local/directory/ on your computer.

      • To copy a file from your Linode to your computer and give it a specific name (in this case, file.txt.backup):

        scp your_linode_username@your_linode_ip:/path/to/your/file.txt /path/to/your/local/directory/file.txt.backup
        
      • To copy a directory from your Linode to your computer:

        scp -r your_linode_username@your_linode_ip:/path/to/your/directory /path/to/your/local/directory
        

        If /path/to/your/local/directory already exists on your computer, then the copied directory will be placed inside /path/to/your/local/directory (i.e. /path/to/your/local/directory/directory).

        If /path/to/your/local/directory does not already exist, then the copied directory will be created with that name.

      For example:

      • Download an NGINX configuration file to your home folder:

        scp your_linode_username@your_linode_ip:/etc/nginx/conf.d/example.com.conf ~/example.com.conf.backup
        
      • Download an Apache configuration file to your home folder:

        scp your_linode_username@your_linode_ip:/etc/apache2/sites-available/example.com.conf ~/example.com.conf.backup
        
      • Copy the entire document root from a web server:

        scp -r your_linode_username@your_linode_ip:/var/www/html/ ~/html_backup
        

      If you intend to repeat this process regularly, consider using rsync to create additional local copies of your data. rsync is capable of performing incremental file copies, which means you do not have to fully transfer each file every time you download your data.

      Using FileZilla

      As an alternative to the command-line, you can download and install an SFTP client. These applications provide a graphical user interface for your Linode’s filesystem.

      FileZilla is a popular free example. Windows and OS X users can download FileZilla here. To install FileZilla on Linux:

      After you’ve installed FileZilla on your computer:

      1. Open FileZilla from your Windows start menu, OS X Launchpad, or the launcher provided by your Linux distribution of choice.

      2. Enter your Linode’s IP address or domain name in the Host field. Enter the account username you wish to connect as in the Username field. Please note that this must be a user account on your Linode; if in doubt, enter root to log in as the root user. Enter the account’s password in the Password field, and enter “22” in the Port field. Click Quickconnect to initiate the file transfer session.

        Quickconnect

      3. If this is the first time you’ve connected to your Linode with an SSH or SFTP program, you’ll receive a warning that the host key is unknown. Place a check mark in the box next to Always trust this host, add this key to the cache. Checking this box prevents further warnings unless the key presented to FileZilla changes; this should only happen if you reinstall the remote server’s operating system.

        Unknown Key

      4. Click the OK button to proceed. You’ll be presented with a split view, with your local filesystem on the left and your Linode’s filesystem on the right. You may transfer files by dragging and dropping them between each side.

      For more information on FileZilla, review our full guide on using the application.

      Using mysqldump to Back Up a Database

      Special care is needed when downloading data from a database. Before it can be downloaded, the data in a database needs to first be dumped to a file. This file can then be transferred just as any other normal file type.

      The mysqldump command’s general syntax is:

      mysqldump -u [username] -p [databaseName] > [filename]-$(date +%F).sql
      
      • mysqldump prompts for a password before it starts the backup process.
      • Depending on the size of the database, it could take a while to complete.
      • The database backup will be created in the directory the command is run.
      • -$(date +%F) adds a timestamp to the filename.

      Example use cases include:

      • Create a backup of an entire Database Management System (DBMS):

        mysqldump --all-databases --single-transaction --quick --lock-tables=false > full-backup-$(date +%F).sql -u root -p
        
      • Back up a specific database. Replace db1 with the name of the database you want to back up:

        mysqldump -u username -p db1 --single-transaction --quick --lock-tables=false > db1-backup-$(date +%F).sql
        
      • Back up a single table from any database. In the example below, table1 is exported from the database db1:

        mysqldump -u username -p --single-transaction --quick --lock-tables=false db1 table1 > db1-table1-$(date +%F).sql
        

      Here’s a breakdown of the mysqldump command options used above:

      • --single-transaction: Issue a BEGIN SQL statement before dumping data from the server.
      • --quick: Enforce dumping tables row by row. This provides added safety for systems with little RAM and/or large databases where storing tables in memory could become problematic.
      • --lock-tables=false: Do not lock tables for the backup session.

      For more information on MySQL database backups, including how to restore the data in a dump file to a MySQL installation, review our guide on the subject. An alternative to using mysqldump is to create physical backups. It’s also possible to backup PostgreSQL databases.

      Download a Disk over SSH

      Downloading your disk will copy a .img file to your computer that encapsulates all of the data that is on your Linode’s disk.

      Prepare the Receiving Computer

      Verify that the receiving computer has SSH installed. (Most Linux/Unix-like systems have it installed by default.) If you’re running Windows locally, you may wish to set up the Cygwin compatibility layer to provide a reasonably complete Unix-like environment. Instructions on setting up Cygwin are located here.

      Start Your Linode in Rescue Mode

      Before you initiate the transfer, start the source Linode in Rescue Mode and start SSH by following these guides:

      1. Start your Linode in Rescue Mode.
      2. Connecting to a Linode Running in Rescue Mode via LISH.
      3. Start the SSH server on your Linode.

      Copy the Disk

      Now that the Linode is running in Rescue Mode, you can transfer the disk from the Linode to the receiving machine over SSH:

      1. Enter the following command on the receiving machine. Replace 123.45.67.89 with the Linode’s IP address and /home/archive/linode.img with the path where you want to store the disk:

        ssh root@123.45.67.89 "dd if=/dev/sda " | dd of=/home/archive/linode.img
        

        Note

        The device /dev/sda is used for Linodes running on top of KVM. If your Linode is still using XEN, then throughout this guide you must use /dev/xvda instead.

      2. The receiving machine will connect to the Linode. Type yes and press Enter to continue connecting:

        The authenticity of host '123.45.67.89 (123.45.67.89)' can't be established.
        RSA key fingerprint is 39:6b:eb:05:f1:28:95:f0:da:63:17:9e:6b:6b:11:4a.
        Are you sure you want to continue connecting (yes/no)? yes
        
      3. Enter the root password for the Linode:

        Warning: Permanently added '123.45.67.89' (RSA) to the list of known hosts.
        root@123.45.67.89's password:
        

        The transfer starts, and you’ll see output similar to the following:

          
                4096000+0 records in
                4096000+0 records out
                2097152000 bytes (2.1 GB) copied, 371.632 seconds, 5.6 MB/s
                4096000+0 records in
                4096000+0 records out
                2097152000 bytes (2.1 GB) copied, 364.002 s, 5.8 MB/s
            
        

        Note

        Copying your disk can take a while. Please be patient. If you have a slow internet connection, add the -C option to the SSH command; this enables gzip compression for data transfer. If you receive a Write failed: Broken pipe error, repeat this process.

      Verify the Disk

      Once the copy has completed, you can verify it by mounting the image on the receiving machine.

      1. Log in to the receiving machine as root by entering the following command and entering the root user’s password:

        su
        
      2. Make a directory on the receiving machine by entering the following command:

        mkdir linode
        
      3. Mount the disk by entering the following command, replacing linode.img with the name of the disk:

        mount -o loop linode.img linode
        
      4. View the directories stored on the disk by entering the following command:

        ls linode/
        

        You should see the directories on the disks, similar to the ones shown below, indicating that everything has transferred:

        bin   dev  home  lost+found  mnt  proc  sbin     srv  tmp  var
        boot  etc  lib   media       opt  root  selinux  sys  usr
        

      Clean Up after Your Download

      After you’ve finished downloading your files or disks, you can optionally delete the restored disks. If you created a new Linode to perform the restore, consider deleting the Linode. As a reminder, billing for that Linode will continue automatically if you do not remove it. If you only power the Linode off, you will still be billed for it.

      Join our Community

      Find answers, ask questions, and help others.

      This guide is published under a CC BY-ND 4.0 license.



      Source link