One place for hosting & domains

      Arrays

      Understanding Arrays in Go


      Introduction

      An array in Go is an ordered sequence of elements that has its capacity defined at creation time. Once an array has allocated its size, the size can no longer be changed. Because the size of an array is static, the data structure only needs to allocate memory once, as opposed to a variable length data structure that must dynamically allocate memory so that it can become larger or smaller in the future.

      Although the fixed length of arrays can make them somewhat rigid to work with, the one-time memory allocation can increase the speed and performance of your program. Because of this, developers typically use arrays when optimizing programs. In Go, slices are the variable length version of arrays. Slices provide more flexibility and constitute what you would think of as arrays in other languages.

      In this article, you will learn how to declare an array, how to call individual elements using indexing, how to slice the array into smaller sets, and the difference between an array and a slice in Go.

      Defining an Array

      Arrays are defined by declaring the size of the array in brackets [ ], followed by the data type of the elements. An array in Go must have all its elements be the same data type. After the data type, you can declare the individual values of the array elements in curly brackets { }.

      The following is the general schema for declaring an array:

      [capacity]data_type{element_values}
      

      Note: It is important to remember that every declaration of a new array creates a distinct type. So, although [2]int and [3]int both have integer elements, their differing capacities make their data types incompatible.

      If you do not declare the values of the array’s elements, the default is zero-valued, which means that the array elements of the array will be empty. For integers, this is represented by 0, and for strings this is represented by an empty string.

      For example, the following array numbers has three integer elements that do not yet have a value:

      var numbers [3]int
      

      If you printed numbers, you would recieve the following output:

      Output

      [0 0 0]

      If you would like to assign the values of the elements when you create the array, place the values in curly brackets. An array of strings with set values looks like this:

      [4]string{"blue coral", "staghorn coral", "pillar coral", "elkhorn coral"}
      

      You can store an array in a variable and print it out:

      coral := [4]string{"blue coral", "staghorn coral", "pillar coral", "elkhorn coral"}
      fmt.Println(coral)
      

      Running a program with the preceding lines would give you the following output:

      Output

      [blue coral staghorn coral pillar coral elkhorn coral]

      Notice that there is no delineation between the elements in the array when it is printed, making it difficult to tell where one element ends and another begins. Because of this, it is sometimes helpful to use the fmt.Printf function instead, which can format strings before printing them to the screen. Provide the %q verb with this command to instruct the function to put quotation marks around the values:

      fmt.Printf("%qn", coral)
      

      This will result in the following:

      Output

      ["blue coral" "staghorn coral" "pillar coral" "elkhorn coral"]

      Now each item is quoted. The n verb instructs to the formatter to add a line return at the end.

      With a general idea of how to declare arrays and what they consist of, you can now move on to learning how to specify elements in an array with indexes.

      Indexing Arrays

      Each element in an array can be called individually through indexing. Each element corresponds to an index number, which is an int value starting from the index number 0 and counting up.

      For the coral array from the earlier example, the index breakdown looks like this:

      “blue coral” “staghorn coral” “pillar coral” “elkhorn coral”
      0 1 2 3

      The first element, the string 'blue coral', starts at index 0, and the list ends at index 3 with the item 'elkhorn coral'.

      You can call a discrete element of the array by referring to its index number in brackets after the variable in which the array is stored:

      fmt.Println(coral[2])
      

      This will print the following:

      Output

      pillar coral

      The index numbers for this array range from 03, so to call any of the elements individually and assign them a value, you could refer to the index numbers like this:

      coral[0] = "blue coral"
      coral[1] = "staghorn coral"
      coral[2] = "pillar coral"
      coral[3] = "elkhorn coral"
      

      If you call the array coral with an index number greater than 3, it will be out of range, and Go will consider the action invalid:

      fmt.Println(coral[22])
      

      Output

      invalid array index 22 (out of bounds for 4-element array)

      When indexing an array, you must always use a positive number. Unlike some languages that let you index backwards with a negative number, doing that in Go will result in an error:

      fmt.Println(coral[-1])
      

      Output

      invalid array index -1 (index must be non-negative)

      Now that you know how to work with individual elements in an array, you can learn how to slice arrays to select a range of elements.

      Slicing Arrays

      By using index numbers to determine beginning and endpoints, you can call a subsection of the values within an array. This is called slicing the array. You can do this by creating a range of index numbers separated by a colon, in the form of [first_index:second_index].

      Let’s say you would like to just print the middle items of coral, without the first and last element. You can do this by creating a slice starting at index 1 and ending just before index 3:

      fmt.Println(coral[1:3])
      

      Running a program with this line would yield the following:

      Output

      [staghorn coral pillar coral]

      When creating a slice, as in [1:3], the first number is where the slice starts (inclusive), and the second number is the sum of the first number and the total number of elements you would like to retrieve:

      array[starting_index : (starting_index + length_of_slice)]
      

      In this instance, you called the second element (or index 1) as the starting point, and called two elements in total. This is how the calculation would look:

      array[1 : (1 + 2)]
      

      Which is how you arrived at this notation:

      coral[1:3]
      

      If you want to set the beginning or end of the array as a starting or end point of the slice, you can omit one of the numbers in the array[first_index:second_index] syntax. For example, if you want to print the first three items of the array coral — which would be "blue coral", "staghorn coral", and "pillar coral" — you can do so by typing:

      fmt.Println(coral[:3])
      

      This will print:

      Output

      [blue coral staghorn coral pillar coral]

      This printed the beginning of the array, stopping right before index 3.

      To include all the items at the end of an array, you would reverse the syntax:

      fmt.Println(coral[1:])
      

      This would give the following:

      Output

      [staghorn coral pillar coral elkhorn coral]

      This section discussed calling individual parts of an array by slicing out subsections. Next, you’ll learn a specific function that Go uses for arrays: len().

      Array Functions

      In Go, len() is a built-in function made to help you work with arrays. Like with strings, you can calculate the length of an array by using len() and passing in the array as a parameter.

      For example, to find how many elements are in the coral array, you would use:

      len(coral)
      

      If you print out the length for the array coral, you’ll receive the following output:

      Output

      4

      This gives the length of the array 4 in the int data type, which is correct because the array coral has four items:

      coral := [4]string{"blue coral", "staghorn coral", "pillar coral", "elkhorn coral"}
      

      If you create an array of integers with more elements, you could use the len() function on this as well:

      numbers := [13]int{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}
      fmt.Println(len(numbers))
      

      This would result in the following output:

      Output

      13

      Although these example arrays have relatively few items, the len() function is especially useful when determining how many elements are in very large arrays.

      Now that you know how to use len() to output the length of arrays, you can learn how arrays differ from another common data structure: slices.

      How Arrays Differ from Slices

      As mentioned before, the primary way in which arrays are different from slices is that the size of an array cannot be modified. This means that while you can change the values of elements in an array, you can’t make the array larger or smaller after it has been defined. A slice, on the other hand, can alter its length.

      Let’s consider your coral array:

      coral := [4]string{"blue coral", "staghorn coral", "pillar coral", "elkhorn coral"}
      

      Say you want to add the item "black coral" to this array. If you try to use the append() function with the array by typing:

      coral = append(coral, "black coral")
      

      You will receive an error as your output:

      Output

      first argument to append must be slice; have [4]string

      If you create an array and decide that you need it to have a variable length, you can convert it to a slice. To convert an array to a slice, use the slicing process you learned in the Slicing Arrays step of this tutorial, except this time select the entire slice by omitting both of the index numbers that would determine the endpoints:

      coral[:]
      

      Keep in mind that you can’t convert the variable coral to a slice itself, since once a variable is defined in Go, its type can’t be changed. To work around this, you can copy the entire contents of the array into a new variable as a slice:

      coralSlice := coral[:]
      

      If you printed coralSlice, you would receive the following output:

      Output

      [blue coral staghorn coral pillar coral elkhorn coral]

      Now, try to use append() with the newly converted slice:

      newSlice := append(coralSlice, "black coral")
      fmt.Printf("%qn", newSlice)
      

      This will output the slice with the added element:

      Output

      ["blue coral" "staghorn coral" "pillar coral" "elkhorn coral" "black coral"]

      Conclusion

      In this tutorial, you learned that the array data type is a sequenced data type with a fixed length, which makes it faster for Go to process at the cost of flexibility. Arrays also can help with communication on a team of developers: When others collaborate with you on your code, your use of arrays will convey to them that you don’t intend for their lengths to be changed.

      With this data type in your tool box, you can now go more in-depth learning the variable length version of this structure: slices.



      Source link

      How To Create RAID Arrays with mdadm on Debian 9


      Introduction

      The mdadm utility can be used to create and manage storage arrays using Linux’s software RAID capabilities. Administrators have great flexibility in coordinating their individual storage devices and creating logical storage devices that have greater performance or redundancy characteristics.

      In this guide, we will go over a number of different RAID configurations that can be set up using a Debian 9 server.

      Prerequisites

      In order to complete the steps in this guide, you should have:

      • A non-root user with sudo privileges on a Debian 9 server: The steps in this guide will be completed with a sudo user. To learn how to set up an account with these privileges, follow our Debian 9 initial server setup guide.
      • A basic understanding of RAID terminology and concepts: While this guide will touch on some RAID terminology in passing, a more complete understanding is very useful. To learn more about RAID and to get a better understanding of what RAID level is right for you, read our introduction to RAID article.
      • Multiple raw storage devices available on your server: We will be demonstrating how to configure various types of arrays on the server. As such, you will need some drives to configure. If you are using DigitalOcean, you can use Block Storage volumes to fill this role. Depending on the array type, you will need at minimum between two to four storage devices. These drives do not need to be formatted prior to following this guide.

      Before we begin, we need to install mdadm, the tool that allows us to set up and manage software RAID arrays in Linux. This is available in Debian’s default repositories.

      Update the local package cache to retrieve an up-to-date list of available packages and then download and install the package:

      • sudo apt update
      • sudo apt install mdadm

      This will install mdadm and all of its dependencies. Verify that the utility is installed by typing:

      Output

      mdadm - v3.4 - 28th January 2016

      The application version should be displayed, indicating that mdadm is installed and ready to use.

      Resetting Existing RAID Devices

      Throughout this guide, we will be introducing the steps to create a number of different RAID levels. If you wish to follow along, you will likely want to reuse your storage devices after each section. This section can be referenced to learn how to quickly reset your component storage devices prior to testing a new RAID level. Skip this section for now if you have not yet set up any arrays.

      Warning: This process will completely destroy the array and any data written to it. Make sure that you are operating on the correct array and that you have copied off any data you need to retain prior to destroying the array.

      Find the active arrays in the /proc/mdstat file by typing:

      Output

      Personalities : [raid0] [linear] [multipath] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid0 sdc[1] sdd[0] 209584128 blocks super 1.2 512k chunks unused devices: <none>

      Unmount the array from the filesystem:

      Then, stop and remove the array by typing:

      • sudo mdadm --stop /dev/md0

      Find the devices that were used to build the array with the following command:

      Warning: Keep in mind that the /dev/sd* names can change any time you reboot! Check them every time to make sure you are operating on the correct devices.

      • lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

      Output

      NAME SIZE FSTYPE TYPE MOUNTPOINT sda 100G disk sdb 100G disk sdc 100G linux_raid_member disk sdd 100G linux_raid_member disk vda 25G disk ├─vda1 24.9G ext4 part / ├─vda14 4M part └─vda15 106M vfat part /boot/efi

      After discovering the devices used to create an array, zero their superblock to remove the RAID metadata and reset them to normal:

      • sudo mdadm --zero-superblock /dev/sdc
      • sudo mdadm --zero-superblock /dev/sdd

      You should remove any of the persistent references to the array. Edit the /etc/fstab file and comment out or remove the reference to your array:

      /etc/fstab

      . . .
      # /dev/md0 /mnt/md0 ext4 defaults,nofail,discard 0 0
      

      Also, comment out or remove the array definition from the /etc/mdadm/mdadm.conf file:

      • sudo nano /etc/mdadm/mdadm.conf

      /etc/mdadm/mdadm.conf

      . . .
      # ARRAY /dev/md0 metadata=1.2 name=mdadmwrite:0 UUID=7261fb9c:976d0d97:30bc63ce:85e76e91
      

      Finally, update the initramfs again so that the early boot process does not try to bring an unavailable array online.

      At this point, you should be ready to reuse the storage devices individually, or as components of a different array.

      Creating a RAID 0 Array

      The RAID 0 array works by breaking up data into chunks and striping it across the available disks. This means that each disk contains a portion of the data and that multiple disks will be referenced when retrieving information.

      • Requirements: minimum of 2 storage devices
      • Primary benefit: Performance
      • Things to keep in mind: Make sure that you have functional backups. A single device failure will destroy all data in the array.

      Identifying the Component Devices

      To get started, find the identifiers for the raw disks that you will be using:

      • lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

      Output

      NAME SIZE FSTYPE TYPE MOUNTPOINT sda 100G disk sdb 100G disk vda 25G disk ├─vda1 24.9G ext4 part / ├─vda14 4M part └─vda15 106M vfat part /boot/efi

      As you can see above, we have two disks without a filesystem, each 100G in size. In this example, these devices have been given the /dev/sda and /dev/sdb identifiers for this session. These will be the raw components we will use to build the array.

      Creating the Array

      To create a RAID 0 array with these components, pass them in to the mdadm --create command. You will have to specify the device name you wish to create (/dev/md0 in our case), the RAID level, and the number of devices:

      • sudo mdadm --create --verbose /dev/md0 --level=0 --raid-devices=2 /dev/sda /dev/sdb

      You can ensure that the RAID was successfully created by checking the /proc/mdstat file:

      Output

      Personalities : [raid0] md0 : active raid0 sdb[1] sda[0] 209584128 blocks super 1.2 512k chunks unused devices: <none>

      As you can see in the highlighted line, the /dev/md0 device has been created in the RAID 0 configuration using the /dev/sda and /dev/sdb devices.

      Creating and Mounting the Filesystem

      Next, create a filesystem on the array:

      • sudo mkfs.ext4 -F /dev/md0

      Create a mount point to attach the new filesystem:

      You can mount the filesystem by typing:

      • sudo mount /dev/md0 /mnt/md0

      Check whether the new space is available by typing:

      • df -h -x devtmpfs -x tmpfs

      Output

      Filesystem Size Used Avail Use% Mounted on /dev/vda1 25G 1003M 23G 5% / /dev/md0 196G 61M 186G 1% /mnt/md0

      The new filesystem is mounted and accessible.

      Saving the Array Layout

      To make sure that the array is reassembled automatically at boot, we will have to adjust the /etc/mdadm/mdadm.conf file. You can automatically scan the active array and append the file by typing:

      • sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

      Afterwards, you can update the initramfs, or initial RAM file system, so that the array will be available during the early boot process:

      Add the new filesystem mount options to the /etc/fstab file for automatic mounting at boot:

      • echo '/dev/md0 /mnt/md0 ext4 defaults,nofail,discard 0 0' | sudo tee -a /etc/fstab

      Your RAID 0 array should now automatically be assembled and mounted each boot.

      Creating a RAID 1 Array

      The RAID 1 array type is implemented by mirroring data across all available disks. Each disk in a RAID 1 array gets a full copy of the data, providing redundancy in the event of a device failure.

      • Requirements: minimum of 2 storage devices
      • Primary benefit: Redundancy
      • Things to keep in mind: Since two copies of the data are maintained, only half of the disk space will be usable

      Identifying the Component Devices

      To get started, find the identifiers for the raw disks that you will be using:

      • lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

      Output

      NAME SIZE FSTYPE TYPE MOUNTPOINT sda 100G disk sdb 100G disk vda 25G disk ├─vda1 24.9G ext4 part / ├─vda14 4M part └─vda15 106M vfat part /boot/efi

      As you can see above, we have two disks without a filesystem, each 100G in size. In this example, these devices have been given the /dev/sda and /dev/sdb identifiers for this session. These will be the raw components we will use to build the array.

      Creating the Array

      To create a RAID 1 array with these components, pass them in to the mdadm --create command. You will have to specify the device name you wish to create (/dev/md0 in our case), the RAID level, and the number of devices:

      • sudo mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sda /dev/sdb

      If the component devices you are using are not partitions with the boot flag enabled, you will likely see the following warning. It is safe to type y to continue:

      Output

      mdadm: Note: this array has metadata at the start and may not be suitable as a boot device. If you plan to store '/boot' on this device please ensure that your boot-loader understands md/v1.x metadata, or use --metadata=0.90 mdadm: size set to 104792064K Continue creating array? y

      The mdadm tool will start to mirror the drives. This can take some time to complete, but the array can be used during this time. You can monitor the progress of the mirroring by checking the /proc/mdstat file:

      Output

      Personalities : [raid0] [linear] [multipath] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid1 sdb[1] sda[0] 104792064 blocks super 1.2 [2/2] [UU] [>....................] resync = 1.5% (1629632/104792064) finish=8.4min speed=203704K/sec unused devices: <none>

      As you can see in the first highlighted line, the /dev/md0 device has been created in the RAID 1 configuration using the /dev/sda and /dev/sdb devices. The second highlighted line shows the progress on the mirroring. You can continue the guide while this process completes.

      Creating and Mounting the Filesystem

      Next, create a filesystem on the array:

      • sudo mkfs.ext4 -F /dev/md0

      Create a mount point to attach the new filesystem:

      You can mount the filesystem by typing:

      • sudo mount /dev/md0 /mnt/md0

      Check whether the new space is available by typing:

      • df -h -x devtmpfs -x tmpfs

      Output

      Filesystem Size Used Avail Use% Mounted on /dev/vda1 25G 1003M 23G 5% / /dev/md0 98G 61M 93G 1% /mnt/md0

      The new filesystem is mounted and accessible.

      Saving the Array Layout

      To make sure that the array is reassembled automatically at boot, we will have to adjust the /etc/mdadm/mdadm.conf file. You can automatically scan the active array and append the file by typing:

      • sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

      Afterwards, you can update the initramfs, or initial RAM file system, so that the array will be available during the early boot process:

      Add the new filesystem mount options to the /etc/fstab file for automatic mounting at boot:

      • echo '/dev/md0 /mnt/md0 ext4 defaults,nofail,discard 0 0' | sudo tee -a /etc/fstab

      Your RAID 1 array should now automatically be assembled and mounted each boot.

      Creating a RAID 5 Array

      The RAID 5 array type is implemented by striping data across the available devices. One component of each stripe is a calculated parity block. If a device fails, the parity block and the remaining blocks can be used to calculate the missing data. The device that receives the parity block is rotated so that each device has a balanced amount of parity information.

      • Requirements: minimum of 3 storage devices
      • Primary benefit: Redundancy with more usable capacity.
      • Things to keep in mind: While the parity information is distributed, one disk's worth of capacity will be used for parity. RAID 5 can suffer from very poor performance when in a degraded state.

      Identifying the Component Devices

      To get started, find the identifiers for the raw disks that you will be using:

      • lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

      Output

      NAME SIZE FSTYPE TYPE MOUNTPOINT sda 100G disk sdb 100G disk sdc 100G disk vda 25G disk ├─vda1 24.9G ext4 part / ├─vda14 4M part └─vda15 106M vfat part /boot/efi

      As you can see above, we have three disks without a filesystem, each 100G in size. In this example, these devices have been given the /dev/sda, /dev/sdb, and /dev/sdc identifiers for this session. These will be the raw components we will use to build the array.

      Creating the Array

      To create a RAID 5 array with these components, pass them in to the mdadm --create command. You will have to specify the device name you wish to create (/dev/md0 in our case), the RAID level, and the number of devices:

      • sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sda /dev/sdb /dev/sdc

      The mdadm tool will start to configure the array (it actually uses the recovery process to build the array for performance reasons). This can take some time to complete, but the array can be used during this time. You can monitor the progress of the mirroring by checking the /proc/mdstat file:

      Output

      Personalities : [raid0] [linear] [multipath] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid5 sdc[3] sdb[1] sda[0] 209584128 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_] [>....................] recovery = 0.9% (1031612/104792064) finish=10.0min speed=171935K/sec unused devices: <none>

      As you can see in the first highlighted line, the /dev/md0 device has been created in the RAID 5 configuration using the /dev/sda, /dev/sdb and /dev/sdc devices. The second highlighted line shows the progress on the build.

      Warning: Due to the way that mdadm builds RAID 5 arrays, while the array is still building, the number of spares in the array will be inaccurately reported. This means that you must wait for the array to finish assembling before updating the /etc/mdadm/mdadm.conf file. If you update the configuration file while the array is still building, the system will have incorrect information about the array state and will be unable to assemble it automatically at boot with the correct name.

      You can continue the guide while this process completes.

      Creating and Mounting the Filesystem

      Next, create a filesystem on the array:

      • sudo mkfs.ext4 -F /dev/md0

      Create a mount point to attach the new filesystem:

      You can mount the filesystem by typing:

      • sudo mount /dev/md0 /mnt/md0

      Check whether the new space is available by typing:

      • df -h -x devtmpfs -x tmpfs

      Output

      Filesystem Size Used Avail Use% Mounted on /dev/vda1 25G 1003M 23G 5% / /dev/md0 196G 61M 186G 1% /mnt/md0

      The new filesystem is mounted and accessible.

      Saving the Array Layout

      To make sure that the array is reassembled automatically at boot, we will have to adjust the /etc/mdadm/mdadm.conf file.

      As mentioned above, before you adjust the configuration, check again to make sure the array has finished assembling. Completing this step before the array is built will prevent the system from assembling the array correctly on reboot:

      Output

      Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] md0 : active raid5 sdc[3] sdb[1] sda[0] 209584128 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU] unused devices: <none>

      The output above shows that the rebuild is complete. Now, we can automatically scan the active array and append the file by typing:

      • sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

      Afterwards, you can update the initramfs, or initial RAM file system, so that the array will be available during the early boot process:

      Add the new filesystem mount options to the /etc/fstab file for automatic mounting at boot:

      • echo '/dev/md0 /mnt/md0 ext4 defaults,nofail,discard 0 0' | sudo tee -a /etc/fstab

      Your RAID 5 array should now automatically be assembled and mounted each boot.

      Creating a RAID 6 Array

      The RAID 6 array type is implemented by striping data across the available devices. Two components of each stripe are calculated parity blocks. If one or two devices fail, the parity blocks and the remaining blocks can be used to calculate the missing data. The devices that receive the parity blocks are rotated so that each device has a balanced amount of parity information. This is similar to a RAID 5 array, but allows for the failure of two drives.

      • Requirements: minimum of 4 storage devices
      • Primary benefit: Double redundancy with more usable capacity.
      • Things to keep in mind: While the parity information is distributed, two disk's worth of capacity will be used for parity. RAID 6 can suffer from very poor performance when in a degraded state.

      Identifying the Component Devices

      To get started, find the identifiers for the raw disks that you will be using:

      • lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

      Output

      NAME SIZE FSTYPE TYPE MOUNTPOINT sda 100G disk sdb 100G disk sdc 100G disk sdd 100G disk vda 25G disk ├─vda1 24.9G ext4 part / ├─vda14 4M part └─vda15 106M vfat part /boot/efi

      As you can see above, we have four disks without a filesystem, each 100G in size. In this example, these devices have been given the /dev/sda, /dev/sdb, /dev/sdc, and /dev/sdd identifiers for this session. These will be the raw components we will use to build the array.

      Creating the Array

      To create a RAID 6 array with these components, pass them in to the mdadm --create command. You will have to specify the device name you wish to create (/dev/md0 in our case), the RAID level, and the number of devices:

      • sudo mdadm --create --verbose /dev/md0 --level=6 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sdd

      The mdadm tool will start to configure the array (it actually uses the recovery process to build the array for performance reasons). This can take some time to complete, but the array can be used during this time. You can monitor the progress of the mirroring by checking the /proc/mdstat file:

      Output

      Personalities : [raid0] [linear] [multipath] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid6 sdd[3] sdc[2] sdb[1] sda[0] 209584128 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU] [>....................] resync = 0.3% (353056/104792064) finish=14.7min speed=117685K/sec unused devices: <none>

      As you can see in the first highlighted line, the /dev/md0 device has been created in the RAID 6 configuration using the /dev/sda, /dev/sdb, /dev/sdc and /dev/sdd devices. The second highlighted line shows the progress on the build. You can continue the guide while this process completes.

      Creating and Mounting the Filesystem

      Next, create a filesystem on the array:

      • sudo mkfs.ext4 -F /dev/md0

      Create a mount point to attach the new filesystem:

      You can mount the filesystem by typing:

      • sudo mount /dev/md0 /mnt/md0

      Check whether the new space is available by typing:

      • df -h -x devtmpfs -x tmpfs

      Output

      Filesystem Size Used Avail Use% Mounted on /dev/vda1 25G 1003M 23G 5% / /dev/md0 196G 61M 186G 1% /mnt/md0

      The new filesystem is mounted and accessible.

      Save the Array Layout

      To make sure that the array is reassembled automatically at boot, we will have to adjust the /etc/mdadm/mdadm.conf file. We can automatically scan the active array and append the file by typing:

      • sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

      Afterwards, you can update the initramfs, or initial RAM file system, so that the array will be available during the early boot process:

      Add the new filesystem mount options to the /etc/fstab file for automatic mounting at boot:

      • echo '/dev/md0 /mnt/md0 ext4 defaults,nofail,discard 0 0' | sudo tee -a /etc/fstab

      Your RAID 6 array should now automatically be assembled and mounted each boot.

      Creating a Complex RAID 10 Array

      The RAID 10 array type is traditionally implemented by creating a striped RAID 0 array composed of sets of RAID 1 arrays. This nested array type gives both redundancy and high performance, at the expense of large amounts of disk space. The mdadm utility has its own RAID 10 type that provides the same type of benefits with increased flexibility. It is not created by nesting arrays, but has many of the same characteristics and guarantees. We will be using the mdadm RAID 10 here.

      • Requirements: minimum of 3 storage devices
      • Primary benefit: Performance and redundancy
      • Things to keep in mind: The amount of capacity reduction for the array is defined by the number of data copies you choose to keep. The number of copies that are stored with mdadm style RAID 10 is configurable.

      By default, two copies of each data block will be stored in what is called the "near" layout. The possible layouts that dictate how each data block is stored are:

      • near: The default arrangement. Copies of each chunk are written consecutively when striping, meaning that the copies of the data blocks will be written around the same part of multiple disks.
      • far: The first and subsequent copies are written to different parts the storage devices in the array. For instance, the first chunk might be written near the beginning of a disk, while the second chunk would be written half way down on a different disk. This can give some read performance gains for traditional spinning disks at the expense of write performance.
      • offset: Each stripe is copied, offset by one drive. This means that the copies are offset from one another, but still close together on the disk. This helps minimize excessive seeking during some workloads.

      You can find out more about these layouts by checking out the "RAID10" section of this man page:

      You can also find this man page online here.

      Identifying the Component Devices

      To get started, find the identifiers for the raw disks that you will be using:

      • lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

      Output

      NAME SIZE FSTYPE TYPE MOUNTPOINT sda 100G disk sdb 100G disk sdc 100G disk sdd 100G disk vda 25G disk ├─vda1 24.9G ext4 part / ├─vda14 4M part └─vda15 106M vfat part /boot/efi

      As you can see above, we have four disks without a filesystem, each 100G in size. In this example, these devices have been given the /dev/sda, /dev/sdb, /dev/sdc, and /dev/sdd identifiers for this session. These will be the raw components we will use to build the array.

      Creating the Array

      To create a RAID 10 array with these components, pass them in to the mdadm --create command. You will have to specify the device name you wish to create (/dev/md0 in our case), the RAID level, and the number of devices.

      You can set up two copies using the near layout by not specifying a layout and copy number:

      • sudo mdadm --create --verbose /dev/md0 --level=10 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sdd

      If you want to use a different layout, or change the number of copies, you will have to use the --layout= option, which takes a layout and copy identifier. The layouts are n for near, f for far, and o for offset. The number of copies to store is appended afterwards.

      For instance, to create an array that has 3 copies in the offset layout, the command would look like this:

      • sudo mdadm --create --verbose /dev/md0 --level=10 --layout=o3 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sdd

      The mdadm tool will start to configure the array (it actually uses the recovery process to build the array for performance reasons). This can take some time to complete, but the array can be used during this time. You can monitor the progress of the mirroring by checking the /proc/mdstat file:

      Output

      Personalities : [raid0] [linear] [multipath] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid10 sdd[3] sdc[2] sdb[1] sda[0] 209584128 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU] [>....................] resync = 1.3% (2832768/209584128) finish=15.8min speed=217905K/sec unused devices: <none>

      As you can see in the first highlighted line, the /dev/md0 device has been created in the RAID 10 configuration using the /dev/sda, /dev/sdb, /dev/sdc and /dev/sdd devices. The second highlighted area shows the layout that was used for this example (2 copies in the near configuration). The third highlighted area shows the progress on the build. You can continue the guide while this process completes.

      Creating and Mounting the Filesystem

      Next, create a filesystem on the array:

      • sudo mkfs.ext4 -F /dev/md0

      Create a mount point to attach the new filesystem:

      You can mount the filesystem by typing:

      • sudo mount /dev/md0 /mnt/md0

      Check whether the new space is available by typing:

      • df -h -x devtmpfs -x tmpfs

      Output

      Filesystem Size Used Avail Use% Mounted on /dev/vda1 25G 1003M 23G 5% / /dev/md0 196G 61M 186G 1% /mnt/md0

      The new filesystem is mounted and accessible.

      Saving the Array Layout

      To make sure that the array is reassembled automatically at boot, we will have to adjust the /etc/mdadm/mdadm.conf file. We can automatically scan the active array and append the file by typing:

      • sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

      Afterwards, you can update the initramfs, or initial RAM file system, so that the array will be available during the early boot process:

      Add the new filesystem mount options to the /etc/fstab file for automatic mounting at boot:

      • echo '/dev/md0 /mnt/md0 ext4 defaults,nofail,discard 0 0' | sudo tee -a /etc/fstab

      Your RAID 10 array should now automatically be assembled and mounted each boot.

      Conclusion

      In this guide, we demonstrated how to create various types of arrays using Linux's mdadm software RAID utility. RAID arrays offer some compelling redundancy and performance enhancements over using multiple disks individually.

      Once you have settled on the type of array needed for your environment and created the device, you will need to learn how to perform day-to-day management with mdadm. Our guide on how to manage RAID arrays with mdadm can help get you started.



      Source link