One place for hosting & domains

      How To Detect and Extract Faces from an Image with OpenCV and Python


      The author selected the Open Internet/Free Speech Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Images make up a large amount of the data that gets generated each day, which makes the ability to process these images important. One method of processing images is via face detection. Face detection is a branch of image processing that uses machine learning to detect faces in images.

      A Haar Cascade is an object detection method used to locate an object of interest in images. The algorithm is trained on a large number of positive and negative samples, where positive samples are images that contain the object of interest. Negative samples are images that may contain anything but the desired object. Once trained, the classifier can then locate the object of interest in any new images.

      In this tutorial, you will use a pre-trained Haar Cascade model from OpenCV and Python to detect and extract faces from an image. OpenCV is an open-source programming library that is used to process images.

      Prerequisites

      Step 1 — Configuring the Local Environment

      Before you begin writing your code, you will first create a workspace to hold the code and install a few dependencies.

      Create a directory for the project with the mkdir command:

      Change into the newly created directory:

      Next, you will create a virtual environment for this project. Virtual environments isolate different projects so that differing dependencies won't cause any disruptions. Create a virtual environment named face_scrapper to use with this project:

      • python3 -m venv face_scrapper

      Activate the isolated environment:

      • source face_scrapper/bin/activate

      You will now see that your prompt is prefixed with the name of your virtual environment:

      Now that you've activated your virtual environment, you will use nano or your favorite text editor to create a requirements.txt file. This file indicates the necessary Python dependencies:

      Next, you need to install three dependencies to complete this tutorial:

      • numpy: numpy is a Python library that adds support for large, multi-dimensional arrays. It also includes a large collection of mathematical functions to operate on the arrays.
      • opencv-utils: This is the extended library for OpenCV that includes helper functions.
      • opencv-python: This is the core OpenCV module that Python uses.

      Add the following dependencies to the file:

      requirements.txt

      numpy 
      opencv-utils
      opencv-python
      

      Save and close the file.

      Install the dependencies by passing the requirements.txt file to the Python package manager, pip. The -r flag specifies the location of requirements.txt file.

      • pip install -r requirements.txt

      In this step, you set up a virtual environment for your project and installed the necessary dependencies. You're now ready to start writing the code to detect faces from an input image in next step.

      Step 2 — Writing and Running the Face Detector Script

      In this section, you will write code that will take an image as input and return two things:

      • The number of faces found in the input image.
      • A new image with a rectangular plot around each detected face.

      Start by creating a new file to hold your code:

      In this new file, start writing your code by first importing the necessary libraries. You will import two modules here: cv2 and sys. The cv2 module imports the OpenCV library into the program, and sys imports common Python functions, such as argv, that your code will use.

      app.py

      import cv2
      import sys
      

      Next, you will specify that the input image will be passed as an argument to the script at runtime. The Pythonic way of reading the first argument is to assign the value returned by sys.argv[1] function to an variable:

      app.py

      ...
      imagePath = sys.argv[1]
      

      A common practice in image processing is to first convert the input image to gray scale. This is because detecting luminance, as opposed to color, will generally yield better results in object detection. Add the following code to take an input image as an argument and convert it to grayscale:

      app.py

      ...
      image = cv2.imread(imagePath)
      gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
      

      The .imread() function takes the input image, which is passed as an argument to the script, and converts it to an OpenCV object. Next, OpenCV's .cvtColor() function converts the input image object to a grayscale object.

      Now that you've added the code to load an image, you will add the code that detects faces in the specified image:

      app.py

      ...
      faceCascade = cv2.CascadeClassifier(cv2.data.haarcascades + "haarcascade_frontalface_default.xml")
      faces = faceCascade.detectMultiScale(
              gray,
              scaleFactor=1.3,
              minNeighbors=3,
              minSize=(30, 30)
      ) 
      
      print("Found {0} Faces!".format(len(faces)))
      
      

      This code will create a faceCascade object that will load the Haar Cascade file with the cv2.CascadeClassifier method. This allows Python and your code to use the Haar Cascade.

      Next, the code applies OpenCV's .detectMultiScale() method on the faceCascade object. This generates a list of rectangles for all of the detected faces in the image. The list of rectangles is a collection of pixel locations from the image, in the form of Rect(x,y,w,h).

      Here is a summary of the other parameters your code uses:

      • gray: This specifies the use of the OpenCV grayscale image object that you loaded earlier.
      • scaleFactor: This parameter specifies the rate to reduce the image size at each image scale. Your model has a fixed scale during training, so input images can be scaled down for improved detection. This process stops after reaching a threshold limit, defined by maxSize and minSize.
      • minNeighbors: This parameter specifies how many neighbors, or detections, each candidate rectangle should have to retain it. A higher value may result in less false positives, but a value too high can eliminate true positives.
      • minSize: This allows you to define the minimum possible object size measured in pixels. Objects smaller than this parameter are ignored.

      After generating a list of rectangles, the faces are then counted with the len function. The number of detected faces are then returned as output after running the script.

      Next, you will use OpenCV's .rectangle() method to draw a rectangle around the detected faces:

      app.py

      ...
      for (x, y, w, h) in faces:
          cv2.rectangle(image, (x, y), (x+w, y+h), (0, 255, 0), 2)
      
      

      This code uses a for loop to iterate through the list of pixel locations returned from faceCascade.detectMultiScale method for each detected object. The rectangle method will take four arguments:

      • image tells the code to draw rectangles on the original input image.
      • (x,y), (x+w, y+h) are the four pixel locations for the detected object. rectangle will use these to locate and draw rectangles around the detected objects in the input image.
      • (0, 255, 0) is the color of the shape. This argument gets passed as a tuple for BGR. For example, you would use (255, 0, 0) for blue. We are using green in this case.
      • 2 is the thickness of the line measured in pixels.

      Now that you've added the code to draw the rectangles, use OpenCV's .imwrite() method to write the new image to your local filesystem as faces_detected.jpg. This method will return true if the write was successful and false if it wasn't able to write the new image.

      app.py

      ...
      status = cv2.imwrite('faces_detected.jpg', image)
      

      Finally, add this code to print the return the true or false status of the .imwrite() function to the console. This will let you know if the write was successful after running the script.

      app.py

      ...
      print ("Image faces_detected.jpg written to filesystem: ",status)
      

      The completed file will look like this:

      app.py

      import cv2
      import sys
      
      imagePath = sys.argv[1]
      
      image = cv2.imread(imagePath)
      gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
      
      faceCascade = cv2.CascadeClassifier(cv2.data.haarcascades + "haarcascade_frontalface_default.xml")
      faces = faceCascade.detectMultiScale(
          gray,
          scaleFactor=1.3,
          minNeighbors=3,
          minSize=(30, 30)
      )
      
      print("[INFO] Found {0} Faces!".format(len(faces)))
      
      for (x, y, w, h) in faces:
          cv2.rectangle(image, (x, y), (x + w, y + h), (0, 255, 0), 2)
      
      status = cv2.imwrite('faces_detected.jpg', image)
      print("[INFO] Image faces_detected.jpg written to filesystem: ", status)
      

      Once you've verified that everything is entered correctly, save and close the file.

      Note: This code was sourced from the publicly available OpenCV documentation.

      Your code is complete and you are ready to run the script.

      Step 3 — Running the Script

      In this step, you will use an image to test your script. When you find an image you'd like to use to test, save it in the same directory as your app.py script. This tutorial will use the following image:

      Input Image of four people looking at phones

      If you would like to test with the same image, use the following command to download it:

      • curl -O https://www.xpresservers.com/wp-content/uploads/2019/03/How-To-Detect-and-Extract-Faces-from-an-Image-with-OpenCV-and-Python.png

      Once you have an image to test the script, run the script and provide the image path as an argument:

      • python app.py path/to/input_image

      Once the script finishes running, you will receive output like this:

      Output

      [INFO] Found 4 Faces! [INFO] Image faces_detected.jpg written to filesystem: True

      The true output tells you that the updated image was successfully written to the filesystem. Open the image on your local machine to see the changes on the new file:

      Output Image with detected faces

      You should see that your script detected four faces in the input image and drew rectangles to mark them. In the next step, you will use the pixel locations to extract faces from the image.

      Step 4 — Extracting Faces and Saving them Locally (Optional)

      In the previous step, you wrote code to use OpenCV and a Haar Cascade to detect and draw rectangles around faces in an image. In this section, you will modify your code to extract the detected faces from the image into their own files.

      Start by reopening the app.py file with your text editor:

      Next, add the highlighted lines under the cv2.rectangle line:

      app.py

      ...
      for (x, y, w, h) in faces:
          cv2.rectangle(image, (x, y), (x + w, y + h), (0, 255, 0), 2)
          roi_color = image[y:y + h, x:x + w] 
          print("[INFO] Object found. Saving locally.") 
          cv2.imwrite(str(w) + str(h) + '_faces.jpg', roi_color) 
      ...
      

      The roi_color object plots the pixel locations from the faces list on the original input image. The x, y, h, and w variables are the pixel locations for each of the objects detected from faceCascade.detectMultiScale method. The code then prints output stating that an object was found and will be saved locally.

      Once that is done, the code saves the plot as a new image using the cv2.imwrite method. It appends the width and height of the plot to the name of the image being written to. This will keep the name unique in case there are multiple faces detected.

      The updated app.py script will look like this:

      app.py

      import cv2
      import sys
      
      imagePath = sys.argv[1]
      
      image = cv2.imread(imagePath)
      gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
      
      faceCascade = cv2.CascadeClassifier(cv2.data.haarcascades + "haarcascade_frontalface_default.xml")
      faces = faceCascade.detectMultiScale(
          gray,
          scaleFactor=1.3,
          minNeighbors=3,
          minSize=(30, 30)
      )
      
      print("[INFO] Found {0} Faces.".format(len(faces)))
      
      for (x, y, w, h) in faces:
          cv2.rectangle(image, (x, y), (x + w, y + h), (0, 255, 0), 2)
          roi_color = image[y:y + h, x:x + w]
          print("[INFO] Object found. Saving locally.")
          cv2.imwrite(str(w) + str(h) + '_faces.jpg', roi_color)
      
      status = cv2.imwrite('faces_detected.jpg', image)
      print("[INFO] Image faces_detected.jpg written to filesystem: ", status)
      

      To summarize, the updated code uses the pixel locations to extract the faces from the image into a new file. Once you have finished updating the code, save and close the file.

      Now that you've updated the code, you are ready to run the script once more:

      • python app.py path/to/image

      You will see the similar output once your script is done processing the image:

      Output

      [INFO] Found 4 Faces. [INFO] Object found. Saving locally. [INFO] Object found. Saving locally. [INFO] Object found. Saving locally. [INFO] Object found. Saving locally. [INFO] Image faces_detected.jpg written to file-system: True

      Depending on how many faces are in your sample image, you may see more or less output.

      Looking at the contents of the working directory after the execution of the script, you'll see files for the head shots of all faces found in the input image.

      Directory Listing

      You will now see head shots extracted from the input image collected in the working directory:

      Extracted Faces

      In this step, you modified your script to extract the detected objects from the input image and save them locally.

      Conclusion

      In this tutorial, you wrote a script that uses OpenCV and Python to detect, count, and extract faces from an input image. You can update this script to detect different objects by using a different pre-trained Haar Cascade from the OpenCV library, or you can learn how to train your own Haar Cascade.



      Source link

      How To Create an Image of Your Linux Environment and Launch It On DigitalOcean


      Introduction

      DigitalOcean’s Custom Images feature allows you to bring your custom Linux and Unix-like virtual disk images from an on-premise environment or another cloud platform to DigitalOcean and use them to start DigitalOcean Droplets.

      As described in the Custom Images documentation, the following image types are supported natively by the Custom Images upload tool:

      Although ISO format images aren’t officially supported, you can learn how to create and upload a compatible image using VirtualBox by following How to Create a DigitalOcean Droplet from an Ubuntu ISO Format Image.

      If you don’t already have a compatible image to upload to DigitalOcean, you can create and compress a disk image of your Unix-like or Linux system, provided it has the prerequisite software and drivers installed.

      We’ll begin by ensuring that our image meets the Custom Images requirements. To do this, we’ll configure the system and install some software prerequisites. Then, we’ll create the image using the dd command-line utility and compress it using gzip. Following that, we’ll upload this compressed image file to DigitalOcean Spaces, from which we can import it as a Custom Image. Finally, we’ll boot up a Droplet using the uploaded image.

      Prerequisites

      If possible, you should use one of the DigitalOcean-provided images as a base, or an official distribution-provided cloud image like Ubuntu Cloud. You can then install software and applications on top of this base image to bake a new image, using tools like Packer and VirtualBox. Many cloud providers and virtualization environments also provide tools to export virtual disks to one of the compatible formats listed above, so, if possible, you should use these to simplify the import process. In the cases where you need to manually create a disk image of your system, you can follow the instructions in this guide. Note that these instructions have only been tested with an Ubuntu 18.04 system, and steps may vary depending on your server’s OS and configuration.

      Before you begin with this tutorial, you should have the following available to you:

      • A Linux or Unix-like system that meets all of the requirements listed in the Custom Images product documentation. For example, your boot disk must have:

        • A max size of 100GB
        • An MBR or GPT partition table with a grub bootloader
        • VirtIO drivers installed
      • A non-root user with administrative privileges available to you on the system you’re imaging. To create a new user and grant it administrative privileges on Ubuntu 18.04, follow our Initial Server Setup with Ubuntu 18.04. To learn how to do this on Debian 9, consult Initial Server Setup with Debian 9.

      • An additional storage device used to store the disk image created in this guide, preferably as large as the disk being copied. This can be an attached block storage volume, an external USB drive, an additional physical disk, etc.

      • A DigitalOcean Space and the s3cmd file transfer utility configured for use with your Space. To learn how to create a Space, consult the Spaces Quickstart. To learn how set up s3cmd for use with your Space, consult the s3cmd 2.x Setup Guide.

      Step 1 — Installing Cloud-Init and Enabling SSH

      To begin, we will install the cloud-Init initialization package. Cloud-init is a set of scripts that runs at boot to configure certain cloud instance properties like default locale, hostname, SSH keys and network devices.

      Steps for installing cloud-init will vary depending on the operating system you have installed. In general, the cloud-init package should be available in your OS’s package manager, so if you’re not using a Debian-based distribution, you should substitute apt in the following steps with your distribution-specific package manager command.

      Installing cloud-init

      In this guide, we’ll use an Ubuntu 18.04 server and so will use apt to download and install the cloud-init package. Note that cloud-init may already be installed on your system (some Linux distributions install cloud-init by default). To check, log in to your server and run the following command:

      If you see the following output, cloud-init has already been installed on your server and you can continue on to configuring it for use with DigitalOcean:

      Output

      usage: /usr/bin/cloud-init [-h] [--version] [--file FILES] [--debug] [--force] {init,modules,single,query,dhclient-hook,features,analyze,devel,collect-logs,clean,status} ... /usr/bin/cloud-init: error: the following arguments are required: subcommand

      If instead you see the following, you need to install cloud-init:

      Output

      cloud-init: command not found

      To install cloud-init, update your package index and then install the package using apt:

      • sudo apt update
      • sudo apt install cloud-init

      Now that we've installed cloud-init, we'll configure it for use with DigitalOcean, ensuring that it uses the ConfigDrive datasource. Cloud-init datasources dictate how cloud-init will search for and update instance configuration and metadata. DigitalOcean Droplets use the ConfigDrive datasource, so we will check that it comes first in the list of datasources that cloud-init searches whenever the Droplet boots.

      Reconfiguring cloud-init

      By default, on Ubuntu 18.04, cloud-init configures itself to use the NoCloud datasource first. This will cause problems when running the image on DigitalOcean, so we need to reconfigure cloud-init to use the ConfigDrive datasource and ensure that cloud-init reruns when the image is launched on DigitalOcean.

      From the command line, navigate to the /etc/cloud/cloud.cfg.d directory:

      • cd /etc/cloud/cloud.cfg.d

      Use the ls command to list the cloud-init config files present in the directory:

      Output

      05_logging.cfg 50-curtin-networking.cfg 90_dpkg.cfg curtin-preserve-sources.cfg README

      Depending on your installation, some of these files may not be present. If present, delete the 50-curtin-networking.cfg file, which configures networking interfaces for your Ubuntu server. When the image is launched on DigitalOcean, cloud-init will run and reconfigure these interfaces automatically, so this file is not necessary. If this file is not deleted, the DigitalOcean Droplet created from this Ubuntu image will have its interfaces misconfigured and won't be accessible from the internet:

      • sudo rm 50-curtin-networking.cfg

      Next, we'll run dpkg-reconfigure cloud-init to remove the NoCloud datasource, ensuring that cloud-init searches for and finds the ConfigDrive datasource used on DigitalOcean:

      • sudo dpkg-reconfigure cloud-init

      You should see the following graphical menu:

      Cloud Init dpkg Menu

      The NoCloud datasource is initially highlighted. Press SPACE to unselect it, then hit ENTER.

      Finally, navigate to /etc/netplan:

      Remove the 50-cloud-init.yaml file, which was generated from the cloud-init networking file we removed previously:

      • sudo rm 50-cloud-init.yaml

      The final step is ensuring that we clean up configuration from the initial cloud-init run so that it reruns when the image is launched on DigitalOcean.

      To do this, run cloud-init clean:

      At this point you've installed and configured cloud-init for use with DigitalOcean. You can now move on to enabling SSH access to your droplet.

      Enable SSH Access

      Once you've installed and configured cloud-init, the next step is to ensure that you have a non-root admin user and password available to you on your machine, as outlined in the prerequisites. This step is essential to diagnose any errors that may arise after uploading your image and launching your Droplet. If a preexisting network configuration or bad cloud-init configuration renders your Droplet inaccesible over the network, you can use this user in combination with the DigitalOcean Droplet Console to access your system and diagnose any problems that may have surfaced.

      Once you've set up your non-root administrative user, the final step is to ensure that you have an SSH server installed and running. SSH often comes preinstalled on many popular Linux distributions. The process for checking whether a service is running will vary depending on your server's operating system.. If you aren't sure of how to do this, consult your OS's documentation on managing services. On Ubuntu, you can verify that SSH is up and running using the following command:

      You should see the following output:

      Output

      ● ssh.service - OpenBSD Secure Shell server Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2018-10-22 19:59:38 UTC; 8 days 1h ago Docs: man:sshd(8) man:sshd_config(5) Process: 1092 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS) Main PID: 1115 (sshd) Tasks: 1 (limit: 4915) Memory: 9.7M CGroup: /system.slice/ssh.service └─1115 /usr/sbin/sshd -D

      If SSH isn't up and running, you can install it using apt (on Debian-based distributions):

      • sudo apt install openssh-server

      By default, the SSH server will start on boot unless configured otherwise. This is desirable when running the system in the cloud, as DigitalOcean can automatically copy in your public key and grant you immediate SSH access to your Droplet after creation.

      Once you've created a non-root administrative user, enabled SSH, and installed cloud-init, you're ready to move on to creating an image of your boot disk.

      Step 2 — Creating Disk Image

      In this step, we'll create a RAW format disk image using the dd command-line utility, and compress it using gzip. We'll then upload the image to DigitalOcean Spaces using s3cmd.

      To begin, log in to your server, and inspect the block device arrangement for your system using lsblk:

      You should see something like the following:

      Output

      NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 12.7M 1 loop /snap/amazon-ssm-agent/495 loop1 7:1 0 87.9M 1 loop /snap/core/5328 vda 252:0 0 25G 0 disk └─vda1 252:1 0 25G 0 part / vdb 252:16 0 420K 1 disk

      In this case, we notice that our main boot disk is /dev/vda, a 25GB disk, and the primary partition, mounted at /, is /dev/vda1. In most cases the disk containing the partition mounted at / will be the source disk to image. We are going to use dd to create an image of /dev/vda.

      At this point, you should decide where you want to store the disk image. One option is to attach another block storage device, preferably as large as the disk you are going to image. You can then save the image to this attached temporary disk and upload it to DigitalOcean Spaces.

      If you have physical access to the server, you can add an additional drive to the machine or attach another storage device, like an external USB disk.

      Another option, which we'll demonstrate in this guide, is copying the image over SSH to a local machine, from which you can upload it to Spaces.

      No matter which method you choose to follow, ensure that the storage device to which you save the compressed image has enough free space. If the disk you're imaging is mostly empty, you can expect the compressed image file to be significantly smaller than the original disk.

      Warning: Before running the following dd command, ensure that any critical applications have been stopped and your system is as quiet as possible. Copying an actively-used disk may result in some corrupted files, so be sure to halt any data-intensive operations and shut down as many running applications as possible.

      Option 1: Creating Image Locally

      The syntax for the dd command we're going to execute looks as follows:

      • dd if=/dev/vda bs=4M conv=sparse | pv -s 25G | gzip > /mnt/tmp_disk/ubuntu.gz

      In this case, we are selecting /dev/vda as the input disk to image, and setting the input/output block sizes to 4MB (from the default 512 bytes). This generally speeds things up a little bit. In addition, we are using the conv=sparse flag to minimize the output file size by skipping over empty space. To learn more about dd's parameters, consult the dd manpage.

      We then pipe the output to the pv pipe viewer utility so we can visually track the progress of the transfer (this pipe is optional, and requires installing pv using your package manager). If you know the size of the initial disk (in this case it's 25G), you can add the -s 25G to the pv pipe to get an ETA for when the transfer will complete.

      We then pipe it all to gzip, and save it in a file called ubuntu.gz on the temporary block storage volume we've attached to the server. Replace /mnt/tmp_disk with the path to the external storage device you've attached to your server.

      Option 2: Creating Image over SSH

      Instead of provisioning additional storage for your remote machine, you can also execute the copy over SSH if you have enough disk space available on your local machine. Note that depending on the bandwidth available to you, this can be slow and you may incur additional costs for data transfer over the network.

      To copy and compress the disk over SSH, execute the following command on your local machine:

      • ssh remote_user@your_server_ip "sudo dd if=/dev/vda bs=4M conv=sparse | gzip -1 -" | dd of=ubuntu.gz

      In this case, we are SSHing into our remote server, executing the dd command there, and piping the output to gzip. We then transfer the gzip output over the network and save it as ubuntu.gz locally. Ensure you have the dd utility available on your local machine before running this command:

      Output

      /bin/dd

      Create the compressed image file using either of the above methods. This may take several hours, depending on the size of the disk you're imaging and the method you're using to create the image.

      Once you've created the compressed image file, you can move on to uploading it to your DigitalOcean Spaces using s3cmd.

      Step 3 — Uploading Image to Spaces and Custom Images

      As described in the prerequisites, you should have s3cmd installed and configured for use with your DigitalOcean Space on the machine containing your compressed image.

      Locate the compressed image file, and upload it to your Space using s3cmd:

      Note: You should replace your_space_name with your Space’s name and not its URL. For example, if your Space’s URL is https://example-space-name.nyc3.digitaloceanspaces.com, then your Space’s name is example-space-name.

      • s3cmd put /path_to_image/ubuntu.gz s3://your_space_name

      Once the upload completes, navigate to your Space using the DigitalOcean Control Panel, and locate the image in the list of files. We will temporarily make the image publicly accessible so that Custom Images can access it and save a copy.

      At the right-hand side of the image listing, click the More drop down menu, then click into Manage Permissions:

      Spaces Object Configuration

      Then, click the radio button next to Public and hit Update to make the image publicly accessible.

      Warning: Your image will temporarily be publicly accessible to anyone with its Spaces path during this process. If you'd like to avoid making your image temporarily public, you can create your Custom Image using the DigitalOcean API. Be sure to set your image to Private using the above procedure after your image has successfully been transferred to Custom Images.

      Fetch the Spaces URL for your image by hovering over the image name in the Control Panel, and hit Copy URL in the window that pops up.

      Now, navigate to Images in the left hand navigation bar, and then Custom Images.

      From here, upload your image using this URL as detailed in the Custom Images Product Documentation.

      You can then create a Droplet from this image. Note that you need to add an SSH key to the Droplet on creation. To learn how to do this, consult How to Add SSH Keys to Droplets.

      Once your Droplet boots up, if you can SSH into it, you've successfully launched your Custom Image as a DigitalOcean Droplet.

      Debugging

      If you attempt to SSH into your Droplet and are unable to connect, ensure that your image meets the listed requirements and has both cloud-init and SSH installed and properly configured. If you still can't access the Droplet, you can attempt to use the DigitalOcean Droplet Console and the non-root user you created earlier to explore the system and debug your networking, cloud-init and SSH configurations. Another way of debugging your image is to use a virtualization tool like Virtualbox to boot up your disk image inside of a virtual machine, and debug your system's configuration from within the VM.

      Conclusion

      In this guide, you've learned how to create a disk image of an Ubuntu 18.04 system using the dd command line utility and upload it to DigitalOcean as a Custom Image from which you can launch Droplets.

      The steps in this guide may vary depending on your operating system, existing hardware, and kernel configuration but, in general, images created from popular Linux distributions should work using this method. Be sure to carefully follow the steps for installing and configuring cloud-init, and ensure that your system meets all the requirements listed in the [prerequisites](todo: link) section above.

      To learn more about Custom Images, consult the offical Custom Images product documentation.



      Source link

      How to Create a DigitalOcean Droplet from an Ubuntu ISO Format Image


      Introduction

      DigitalOcean’s Custom Images feature allows you to bring your virtual disk images from an on-premise environment or another cloud platform to DigitalOcean and use them to start DigitalOcean Droplets.

      As described in the Custom Images documentation, the following image types are supported natively by the Custom Images upload tool:

      ISO is another popular image format which you may want to use with Custom Images. ISO images are frequently provided by Linux distributions as a convenient method for installing Linux. Unfortunately, ISO images aren’t currently supported by the upload tool, although support is planned for the end of 2018.

      In this tutorial, we’ll demonstrate how to use the free and open-source VirtualBox virtualization tool to create a DigitalOcean-compatible VDI image (VirtualBox Disk Image) from an Ubuntu 18.04 ISO. The steps in this guide can be adapted to work with your preferred distribution’s ISO images.

      Prerequisites

      Before you begin, you’ll need the following available to you:

      If you’re adapting these steps for another distribution’s ISO and your image does not have cloud-init installed and configured, you must install and configure it manually after installing the OS.

      Once you have these prerequisites available to you, you’re ready to begin with this guide.

      Step 1 — Installing VirtualBox and Creating a Virtual Machine

      The tool we’ll use to convert the ISO-format image in this guide is VirtualBox, a free and open-source virtualizer for x86 hardware. By default, VirtualBox uses a GUI, which we’ll use to create the VDI image in this guide.

      To begin, download and install VirtualBox from the downloads page. Follow the appropriate link in the VirtualBox 5.2.20 platform packages section depending on your host operating system. In this guide, we’ll be using an OSX system, so we’ll download and install VirtualBox using the provided DMG.

      Once you’ve installed VirtualBox, open the application.

      You should see the following welcome screen:

      VirtualBox Welcome Screen

      Click on New to begin creating your Ubuntu virtual machine.

      The following window should pop up, allowing you to name your virtual machine (VM) and select its OS:

      Name Virtual Machine Window

      In this tutorial, we’ll name our VM Ubuntu 18.04, but feel free to give the VM a more descriptive name.

      For Type, select Linux, and for Version, select Ubuntu (64-bit). Then, hit Continue.

      The following screen should appear, allowing you to specify how much memory to allocate to your virtual machine:

      Allocate Memory Window

      Unless you have a more complex use case, 1024 MB should be enough memory for your virtual machine. If you need to adjust memory size, enter the amount of memory to be allocated to the VM, then hit Continue.

      You should see the following screen:

      Create Hard Disk Window

      This window allows you to create a virtual hard disk for your VM. This virtual hard disk is the image that you’ll upload to DigitalOcean in a later step. The Ubuntu operating system will be installed from the ISO you downloaded to this virtual hard disk. Make sure Create a virtual hard disk now is selected, and hit Create.

      The following Hard disk file type window should appear, allowing you to select the format you’d like to use for your image:

      Select Hard Disk Type Window

      All three types are supported by DigitalOcean Custom Images, so unless you have a strong preference, select VDI (VirtualBox Disk Image). Hit Continue.

      You should then see the following window:

      Hard Disk Options

      This window allows you to choose between a Dynamically allocated or Fixed size hard disk file. We’ll use the default Dynamically allocated option and allow the file to grow as we install the Ubuntu OS and packages. Hit Continue.

      The next window allows you to name your hard disk file (as well as choose the path to which it will be saved), and specify its maximum size:

      Hard Disk Size

      Be sure to give yourself enough disk space to install the operating system as well as additional packages you may need. The default 10 GB should be fine for most purposes, but if you anticipate installing a large number of packages or storing a lot of data in the image, you should bump this up to your anticipated disk usage.

      Once you’ve selected the size of the virtual hard disk, hit Create.

      At this point, you’ll be returned to the initial welcome screen, where you’ll see the virtual machine you just created:

      VM Welcome Screen

      We can now begin installing Ubuntu onto the virtual machine.

      Step 2 — Installing Ubuntu 18.04 onto the Virtual Machine

      In this step we’ll install and configure the Ubuntu operating system onto our virtual machine.

      To begin, from the VirtualBox welcome screen, select your virtual machine, and hit the Start button in the toolbar.

      You should see the following virtual machine window, prompting you to select the ISO file from which you’ll boot the system:

      Select ISO

      Select the Ubuntu 18.04 Server ISO you downloaded, and hit Start.

      In the VM, the Ubuntu installer will begin booting from the ISO, and you should be brought to the following menu:

      Ubuntu Select Language

      Choose your preferred language using the arrow keys, and hit ENTER to continue.

      You should then see the following Keyboard configuration screen:

      Ubuntu Keyboard Config

      Choose your preferred keyboard configuration, select Done, and hit ENTER.

      Next, you’ll be brought to the following installer selection screen:

      Ubuntu Installer Selection

      Select Install Ubuntu, and hit ENTER.

      The following Network connections screen should appear:

      Ubuntu Network connections

      This screen allows you to configure the network interfaces for your Ubuntu server. Since we’re performing the installation on a virtual machine, we’ll just use the default option as the configured interface will be overwritten when we launch the image on the DigitalOcean platform.

      Select Done and hit ENTER.

      You’ll then be brought to the following Configure proxy screen:

      Ubuntu Configure Proxy

      If you require a proxy, enter it here. Then, select Done, and hit ENTER.

      The next screen will allow you to choose an Ubuntu archive mirror:

      Ubuntu Archive Mirror

      Unless you require a specific mirror, the default should be fine here. Select Done and hit ENTER.

      Next, you’ll be prompted to partition your virtual disk:

      Ubuntu Partition Disk

      Unless you’d like to set up Logical Volume Manager (LVM) or manually partition the virtual disk, select Use An Entire Disk to use the entire attached virtual disk, and hit ENTER.

      The following screen allows you to select the virtual disk that will be partitioned:

      Ubuntu Filesystem setup

      As described in the prompt text, the installer will create a partition for the bootloader, and use the remaining virtual disk space to create an ext4 partition to which the Ubuntu OS will be installed.

      Select the attached virtual disk and hit ENTER.

      The following screen displays a summary of the filesystem installer options before partitioning:

      Ubuntu Filesystem Summary

      The ext4 partition will be mounted to /, and a second partition (1 MB) will be created for the GRUB bootloader. Once you’ve gone over and confirmed the partitioning scheme for your virtual disk, select Done and hit ENTER.

      In the confirmation screen that appears, select Continue and hit ENTER.

      The next screen will allow you to configure the system hostname, as well as an Ubuntu user:

      Ubuntu Create User

      Note that as you fill out this screen, the installer will continue copying files to the virtual disk in the background.

      In this tutorial, we’ll create a user named sammy and call our server ubuntu. The server name will likely be overwritten when this image is run on the DigitalOcean platform, so feel free to give it a temporary name here.

      You can upload your SSH keys to DigitalOcean and automatically embed them into created Droplets, so for now we won’t Import SSH identity. To learn how to upload your SSH keys to DigitalOcean, consult the Droplet Product Documentation.

      Once you’ve filled in all the required fields, the prompt should look something like this:

      Ubuntu Profile Complete

      Select Done and hit ENTER.

      The next screen will prompt you to select popular snaps for your Ubuntu server. Snaps are prepackaged bundles of software that contain an application, its dependencies, and configuration. To learn more about snaps, consult the Snap Documentation.

      Ubuntu Select Snaps

      In this guide we won’t install any snaps and will manually install packages in a later step. If you’d like to install a snap, select or deselect it using SPACE and scroll down to Done. Then, hit ENTER.

      Regardless of your selection in the snap screen, you’ll then be brought to an installation progress and summary screen:

      Ubuntu Install Progress

      Once the installation completes, select Reboot Now and hit ENTER.

      The installer will shut down and prompt you to remove the installation medium (in this case this is the ISO image we selected earlier). In most cases, the ISO will be detached automatically upon reboot, so you can simply hit ENTER.

      To double check, in the VirtualBox GUI menu, navigate to Devices, and then Optical Drives. If the Remove disk from virtual drive option is available to you, click on it to detach the ISO from the virtual machine. Then, back in the virtual machine window, hit ENTER.

      The system will reboot in the virtual machine, this time from the virtual disk to which we installed Ubuntu.

      Since cloud-init is installed by default on Ubuntu 18.04 Server, the first time Ubuntu boots, cloud-init will run and configure itself. In the virtual machine window, you should see some cloud-init log items and have a prompt available to you. Hit ENTER.

      You can then log in to your Ubuntu server using the user you created in the installer.

      Enter your username and hit ENTER, then enter your password and hit ENTER.

      You should now have access to a command prompt, indicating that you’ve successfully completed the Ubuntu 18.04 installation, and are now logged in as the user you created previously.

      In the next step of this guide, we’ll reconfigure cloud-init and set it up to run when the Ubuntu image is launched as a Droplet on the DigitalOcean platform.

      Step 3 — Reconfiguring cloud-init

      Now that we’ve installed Ubuntu 18.04 to a virtual disk and have the system up and running, we need to reconfigure cloud-init to use the appropriate datasource for the DigitalOcean platform. A cloud-init datasource is a source of config data for cloud-init that typically consists of userdata (like shell scripts) or server metadata, like hostname, instance-id, etc. To learn more about cloud-init datasources, consult the official cloud-init docs.

      By default, on Ubuntu 18.04, cloud-init configures itself to use the DataSourceNoCloud datasource. This will cause problems when running the image on DigitalOcean, so we need to reconfigure cloud-init to use the ConfigDrive datasource and ensure that cloud-init reruns when the image is launched on DigitalOcean.

      To begin, ensure that you’ve started your Ubuntu 18.04 virtual machine and have logged in as the user you created earlier.

      From the command line, navigate to the /etc/cloud/cloud.cfg.d directory:

      • cd /etc/cloud/cloud.cfg.d

      Use the ls command to list the cloud-init config files present in the directory:

      Output

      05_logging.cfg 50-curtin-networking.cfg 90_dpkg.cfg curtin-preserve-sources.cfg README

      First, delete the 50-curtin-networking.cfg file, which configures networking interfaces for your Ubuntu server. When the image is launched on DigitalOcean, cloud-init will run and reconfigure these interfaces automatically. If this file is not deleted, the DigitalOcean Droplet created from this Ubuntu image will have its interfaces misconfigured and won't be accessible from the internet.

      • sudo rm 50-curtin-networking.cfg

      Next, we'll run dpkg-reconfigure cloud-init to remove the NoCloud datasource, ensuring that cloud-init searches for and finds the ConfigDrive datasource used on DigitalOcean:

      • sudo dpkg-reconfigure cloud-init

      You should see the following graphical menu:

      Cloud Init dpkg Menu

      The NoCloud datasource is initially highlighted. Press SPACE to unselect it, then hit ENTER.

      Finally, navigate to /etc/netplan:

      Remove the 50-cloud-init.yaml file (this was generated from the cloud-init networking file we removed earlier):

      • sudo rm 50-cloud-init.yaml

      The final step is ensuring that we clean up configuration from the initial cloud-init run so that it reruns when the image is launched on DigitalOcean.

      To do this, run cloud-init clean:

      At this point, your image is ready to be launched on the DigitalOcean platform. You can install additional packages and software into your image. Once you're done, shutdown your virtual machine:

      We can now move on to uploading and launching this custom image on the DigitalOcean platform.

      Step 4 — Uploading Custom Image and Creating Droplet

      Now that we've created an Ubuntu 18.04 VDI image and configured it for use on DigitalOcean, we can upload it using the Custom Images upload tool.

      On macOS, the Ubuntu virtual disk image we created and configured will be located by default at ~/VirtualBox VMs/your_VM_name/your_virtual_disk_name.vdi. This path may vary slightly depending on the OS you're using with VirtualBox.

      Before we upload the image, we'll compress it to speed up the file transfer to DigitalOcean.

      On your host OS (not inside the virtual machine), navigate to the directory containing your VDI image file:

      • cd ~/VirtualBox VMs/Ubuntu 18.04/

      Now, use gzip to compress the file:

      • gzip < Ubuntu 18.04.vdi > Ubuntu 18.04.gz

      In this command we pipe the source Ubuntu 18.04.vdi file into gzip, specifying as output the Ubuntu 18.04.gz compressed file.

      Once gzip finishes compressing your file, upload the .gz file to DigitalOcean, following instructions in the Custom Images Quickstart.

      You should now be able to create and use Droplets from your custom Ubuntu 18.04 Server image.

      Conclusion

      In this tutorial, we learned how to create a custom VDI image from a vanilla Ubuntu 18.04 ISO using the VirtualBox virtualization tool. We adjusted cloud-init so it can properly configure Droplet networking on DigitalOcean, and finally compressed and uploaded the image using the Custom Images upload tool.

      You can adjust the steps in this tutorial to work with your preferred Linux distribution’s ISO images. Ensure that you have an SSH server installed and configured to start on boot, and that cloud-init has been installed and properly configured to use the ConfigDrive datasource. Finally, ensure that any stale networking configuration files have been purged.

      You may also wish to use a tool like Packer to automate the creation of your machine images.

      To learn more about DigitalOcean Custom Images, consult the Custom Images product docs and launch blog post.



      Source link