Difference between revisions of "Software RAID on Linux"
m |
m (→Software RAID on 3.x Linux) |
||
(14 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
+ | [[Software RAID]] is performed via the Linux kernel. No RAID controller is required on the computer. If the computer has a Linux compatible RAID controller then hardware RAID can be used. [[Hardware RAID]] is not discussed here. | ||
+ | |||
+ | mdadm is a Linux utility used to manage and monitor software RAID devices. It is used in modern GNU/Linux distributions in place of older software RAID utilities such as raidtools2 or raidtools. | ||
+ | |||
+ | == Software RAID on 3.x Linux== | ||
+ | === the /boot filesystem === | ||
+ | In the past the /boot filesystem must be used either without md support, or else with RAID1. With modern kernels the it is possible to load the MD support as a kernel module through the initramfs mechanism. This approach allows the /boot filesystem to be inside any RAID system without the need of a complex manual configuration. | ||
+ | * In the past you couldn't put /boot inside the RAID array, now you can. | ||
+ | |||
+ | So you have a couple options in order to have a setup where each drive can be bootable. One is to have a /boot partition on each drive, or the other involves having /boot within a RAID array. | ||
+ | |||
+ | === Mdadm === | ||
+ | The mdadm utility can be used to create and manage storage arrays. | ||
+ | |||
+ | === Example: 3-way RAID-1 array === | ||
+ | The /boot partition is stored at /dev/md0. This installs GRUB to each disk, so that if one disk fails, you can boot off one of the other disks. | ||
+ | # grub | ||
+ | grub> find /grub/stage1 | ||
+ | (hd0,0) | ||
+ | (hd1,0) | ||
+ | (hd2,0) | ||
+ | grub> device (hd0) /dev/sda | ||
+ | grub> root (hd0,0) | ||
+ | grub> setup (hd0) | ||
+ | grub> device (hd0) /dev/sdb | ||
+ | grub> root (hd0,0) | ||
+ | grub> setup (hd0) | ||
+ | grub> device (hd0) /dev/sdc | ||
+ | grub> root (hd0,0) | ||
+ | grub> setup (hd0) | ||
+ | grub> quit | ||
+ | |||
+ | == Legacy Linux Software RAID == | ||
<big>'''Linux Software RAID'''</big> - An unorganized and random comments page which may or may not be helpful to you. *Created: Mon Mar 29 18:28:01 CST 2004 | <big>'''Linux Software RAID'''</big> - An unorganized and random comments page which may or may not be helpful to you. *Created: Mon Mar 29 18:28:01 CST 2004 | ||
− | == | + | For the most part software RAID functions under linux now as it did ten years ago. However, since the information in this section was collected over ten years ago it will be considered outdated. |
+ | |||
+ | === Software RAID 1 Example === | ||
+ | For this example an older 32-bit Pentium computer is used as a file server with two large capacity drives configured to RAID Level 1: Drive Mirroring. | ||
+ | |||
+ | Trisquel Linux is a lightweight Debian based distribution. Booting from the Live CD load the Trisuel environment. Installation can be performed here only after the drives are properly configured for RAID. Partition the drives and create the RAID array before installing the Linux distribution. | ||
+ | |||
+ | Each drive is the same size. Each drive needs to have identical partitions created. Using fdisk create a small partition of less than 1GB and use the remaining space for the second partition. Due this for each drive. The starting block will likely be at 2048, which is fine. Ensure that you use the same starting block and ending block on both. | ||
+ | |||
+ | When specifying the ending block in fdisk new partition creation, you can specify the starting block as 2048 and then for the ending block specify a size in megabytes. Use the format beginning with the + sign. For 840MB enter "+840" when it asks for the ending block. | ||
+ | |||
+ | Step Summary: | ||
+ | #Create primary partition at the beginning of each of the two hard drives and of a small size such as 1GB or less. | ||
+ | #Create another primary partition using the remaining disk space on each of the drives. | ||
+ | #Use fdisk to convert partitions to a type needed for building the RAID array | ||
+ | #Install RAID manager: mdadm apt-get install mdadm | ||
+ | #The first partition will be used to mount /boot | ||
+ | #The second partition will be used for SWAP space and / root. | ||
+ | #Install Linux on the RAID partitions | ||
+ | |||
+ | When specifying the ending block in fdisk new partition creation, you can specify the starting block as 2048 and then for the ending block specify a size in megabytes. Use the format beginning with the + sign. For 840MB enter "+840" when it asks for the ending block. | ||
+ | |||
+ | If you have 2 drives, /dev/sda and /dev/sdb | ||
+ | fdisk /dev/sda | ||
+ | |||
+ | Create Partitions with fdisk | ||
+ | |||
+ | * d - delete any existing partitions | ||
+ | * n - create a new partition | ||
+ | |||
+ | fdisk /dev/sdb | ||
+ | |||
+ | Create Partitions with fdisk | ||
+ | |||
+ | * d - delete any existing partitions | ||
+ | * n - create a new partition | ||
+ | |||
+ | Once again using fdisk go back into each drive and change the "type" of the partitions to | ||
+ | |||
+ | |||
+ | Now use fdisk to convert partitions to a type needed for building the RAID array | ||
+ | |||
+ | fdisk /dev/sda | ||
+ | |||
+ | * t - change the partition type to "fd". You can get a list of available types | ||
+ | |||
+ | |||
+ | === Software RAID 5 Example === | ||
/boot on the boot partition cannot be RAID 5. This is a limitation in Linux software raid. For this example I have 3 IDE harddrives set up for RAID level five and a seperate raid partition for the following: | /boot on the boot partition cannot be RAID 5. This is a limitation in Linux software raid. For this example I have 3 IDE harddrives set up for RAID level five and a seperate raid partition for the following: | ||
Line 22: | Line 102: | ||
hda6 + hdb6 + hdc6 assigned to md1 (swap) | hda6 + hdb6 + hdc6 assigned to md1 (swap) | ||
− | == (Q). Did everything load ok with RAID when I booted? == | + | === (Q). Did everything load ok with RAID when I booted? === |
Take a look at the boot messages and see for yourself. Type the following command at the console: | Take a look at the boot messages and see for yourself. Type the following command at the console: | ||
'dmesg|less' | 'dmesg|less' | ||
− | == (Q). Hows RAID doing now? or Has a drive died while its running? == | + | === (Q). Hows RAID doing now? or Has a drive died while its running? === |
Take a look at the raid status by typing the following command at the console: | Take a look at the raid status by typing the following command at the console: | ||
− | 'cat /proc/mdstat' | + | <nowiki>'cat /proc/mdstat'</nowiki> |
− | == (Q). What does it mean when I see "kicking hda1" or "kicking" whatever in the | + | === (Q). What does it mean when I see "kicking hda1" or "kicking" whatever in the dmesg and RAID / the system will not start? === |
− | dmesg and RAID / the system will not start? == | + | |
Looks like you have a bad drive or the drive is fine but RAID seems to think | Looks like you have a bad drive or the drive is fine but RAID seems to think | ||
Line 48: | Line 127: | ||
You can usually tell what drive belongs to what array by looking at your RAID | You can usually tell what drive belongs to what array by looking at your RAID | ||
− | configuration file, which is /etc/raidtab. | + | configuration file, which is /etc/raidtab. |
− | == (*). Useful command summary: == | + | === (*). Useful command summary: === |
vi /etc/raidtab | vi /etc/raidtab | ||
Line 61: | Line 140: | ||
lsraid -A -d /dev/hdc2 | lsraid -A -d /dev/hdc2 | ||
− | == (man). lsraid usage examples == | + | === (man). lsraid usage examples === |
− | * lsraid -A -a /dev/md0 | + | * '''lsraid -A -a /dev/md0''' |
:Display a short listing of the md0 device. | :Display a short listing of the md0 device. | ||
− | * lsraid -A -d /dev/sda1 | + | * '''lsraid -A -d /dev/sda1''' |
:Display a short listing of the array that sda1 belongs to. | :Display a short listing of the array that sda1 belongs to. | ||
− | * lsraid -A -f -a /dev/md0 | + | * '''lsraid -A -f -a /dev/md0''' |
:Display the failed devices belonging to the md0 device. | :Display the failed devices belonging to the md0 device. | ||
− | * lsraid -D -l -a /dev/md0 | + | * '''lsraid -D -l -a /dev/md0''' |
:Display a long dump of the on-disk md superblock of every disk in | :Display a long dump of the on-disk md superblock of every disk in | ||
:md0. | :md0. | ||
− | * lsraid -D -a /dev/md0 -d /dev/sda1 | + | * '''lsraid -D -a /dev/md0 -d /dev/sda1''' |
:Display a short discription of the disks in md0 as well as a short | :Display a short discription of the disks in md0 as well as a short | ||
:description of the disk sda1. sda1 will only be described once if | :description of the disk sda1. sda1 will only be described once if | ||
:it belongs to md0. | :it belongs to md0. | ||
− | * lsraid -R -a /dev/md0 -a /dev/md1 -a /dev/md2 | + | * '''lsraid -R -a /dev/md0 -a /dev/md1 -a /dev/md2''' |
:Display a description of the arrays in an output format suitable | :Display a description of the arrays in an output format suitable | ||
:for using in raidtab(5) files. Note that if md0 and md1 are raid0 | :for using in raidtab(5) files. Note that if md0 and md1 are raid0 | ||
Line 87: | Line 166: | ||
:will output the information in the correct order. | :will output the information in the correct order. | ||
− | * lsraid -R -p | + | * '''lsraid -R -p''' |
:Scan all block devices in /proc/partitions and display all discov- | :Scan all block devices in /proc/partitions and display all discov- | ||
:ered md devices in a format suitable for using in raidtab(5) files. | :ered md devices in a format suitable for using in raidtab(5) files. | ||
+ | |||
+ | === Command Reference by Example === | ||
+ | |||
+ | vi /etc/raidtab | ||
+ | |||
+ | dmesg | ||
+ | |||
+ | cat /proc/mdstat | ||
+ | |||
+ | raidhotadd /dev/mdN /dev/hdXN | ||
+ | |||
+ | raidhotremove | ||
+ | |||
+ | lsraid -a /dev/mdN | ||
+ | |||
+ | lsraid -A -d /dev/hdcN | ||
+ | |||
+ | raidstop | ||
+ | |||
+ | raidsetfaulty | ||
+ | |||
+ | mkraid -f (dangerous!) | ||
+ | |||
+ | badblocks | ||
+ | |||
+ | mount /proc /proc -t proc | ||
| |
Latest revision as of 10:47, 3 May 2019
Software RAID is performed via the Linux kernel. No RAID controller is required on the computer. If the computer has a Linux compatible RAID controller then hardware RAID can be used. Hardware RAID is not discussed here.
mdadm is a Linux utility used to manage and monitor software RAID devices. It is used in modern GNU/Linux distributions in place of older software RAID utilities such as raidtools2 or raidtools.
Contents
- 1 Software RAID on 3.x Linux
- 2 Legacy Linux Software RAID
- 2.1 Software RAID 1 Example
- 2.2 Software RAID 5 Example
- 2.3 (Q). Did everything load ok with RAID when I booted?
- 2.4 (Q). Hows RAID doing now? or Has a drive died while its running?
- 2.5 (Q). What does it mean when I see "kicking hda1" or "kicking" whatever in the dmesg and RAID / the system will not start?
- 2.6 (*). Useful command summary:
- 2.7 (man). lsraid usage examples
- 2.8 Command Reference by Example
Software RAID on 3.x Linux
the /boot filesystem
In the past the /boot filesystem must be used either without md support, or else with RAID1. With modern kernels the it is possible to load the MD support as a kernel module through the initramfs mechanism. This approach allows the /boot filesystem to be inside any RAID system without the need of a complex manual configuration.
- In the past you couldn't put /boot inside the RAID array, now you can.
So you have a couple options in order to have a setup where each drive can be bootable. One is to have a /boot partition on each drive, or the other involves having /boot within a RAID array.
Mdadm
The mdadm utility can be used to create and manage storage arrays.
Example: 3-way RAID-1 array
The /boot partition is stored at /dev/md0. This installs GRUB to each disk, so that if one disk fails, you can boot off one of the other disks.
# grub grub> find /grub/stage1 (hd0,0) (hd1,0) (hd2,0) grub> device (hd0) /dev/sda grub> root (hd0,0) grub> setup (hd0) grub> device (hd0) /dev/sdb grub> root (hd0,0) grub> setup (hd0) grub> device (hd0) /dev/sdc grub> root (hd0,0) grub> setup (hd0) grub> quit
Legacy Linux Software RAID
Linux Software RAID - An unorganized and random comments page which may or may not be helpful to you. *Created: Mon Mar 29 18:28:01 CST 2004
For the most part software RAID functions under linux now as it did ten years ago. However, since the information in this section was collected over ten years ago it will be considered outdated.
Software RAID 1 Example
For this example an older 32-bit Pentium computer is used as a file server with two large capacity drives configured to RAID Level 1: Drive Mirroring.
Trisquel Linux is a lightweight Debian based distribution. Booting from the Live CD load the Trisuel environment. Installation can be performed here only after the drives are properly configured for RAID. Partition the drives and create the RAID array before installing the Linux distribution.
Each drive is the same size. Each drive needs to have identical partitions created. Using fdisk create a small partition of less than 1GB and use the remaining space for the second partition. Due this for each drive. The starting block will likely be at 2048, which is fine. Ensure that you use the same starting block and ending block on both.
When specifying the ending block in fdisk new partition creation, you can specify the starting block as 2048 and then for the ending block specify a size in megabytes. Use the format beginning with the + sign. For 840MB enter "+840" when it asks for the ending block.
Step Summary:
- Create primary partition at the beginning of each of the two hard drives and of a small size such as 1GB or less.
- Create another primary partition using the remaining disk space on each of the drives.
- Use fdisk to convert partitions to a type needed for building the RAID array
- Install RAID manager: mdadm apt-get install mdadm
- The first partition will be used to mount /boot
- The second partition will be used for SWAP space and / root.
- Install Linux on the RAID partitions
When specifying the ending block in fdisk new partition creation, you can specify the starting block as 2048 and then for the ending block specify a size in megabytes. Use the format beginning with the + sign. For 840MB enter "+840" when it asks for the ending block.
If you have 2 drives, /dev/sda and /dev/sdb
fdisk /dev/sda
Create Partitions with fdisk
- d - delete any existing partitions
- n - create a new partition
fdisk /dev/sdb
Create Partitions with fdisk
- d - delete any existing partitions
- n - create a new partition
Once again using fdisk go back into each drive and change the "type" of the partitions to
Now use fdisk to convert partitions to a type needed for building the RAID array
fdisk /dev/sda
- t - change the partition type to "fd". You can get a list of available types
Software RAID 5 Example
/boot on the boot partition cannot be RAID 5. This is a limitation in Linux software raid. For this example I have 3 IDE harddrives set up for RAID level five and a seperate raid partition for the following:
/boot assigned to md0 with RAID level 1 /tmp assigned to md2 with RAID level 5 / assigned to md3 with RAID level 5 /home assigned to md4 with RAID level 5
and the linux swap partition on md1 RAID level 5.
Three harddrives are hda, hdb, and hdc.
md0 is a raid partition, consisting of hda1, hdb1, and hdc1. A complete table of raid partitions with drive assignments is as follows:
hda1 + hdb1 + hdc1 assigned to md0 (/boot) hda2 + hdb2 + hdc2 assigned to md4 (/home) hda3 + hdb3 + hdc3 assigned to md3 (/) hda5 + hdb5 + hdc5 assigned to md2 (/tmp) hda6 + hdb6 + hdc6 assigned to md1 (swap)
(Q). Did everything load ok with RAID when I booted?
Take a look at the boot messages and see for yourself. Type the following command at the console:
'dmesg|less'
(Q). Hows RAID doing now? or Has a drive died while its running?
Take a look at the raid status by typing the following command at the console:
'cat /proc/mdstat'
(Q). What does it mean when I see "kicking hda1" or "kicking" whatever in the dmesg and RAID / the system will not start?
Looks like you have a bad drive or the drive is fine but RAID seems to think there is a problem. See which drive was kicked and try using the following command to add it back into the array: 'raidhotadd /dev/md0 /dev/hda1' replacing md0 with the array that the drive was kicked from and hda1 with the actual drive that was kicked from that array. Be careful not to accidently use raidhotadd to add a drive partition into the wrong array. Linux/RAID will let you do this but you will regret it.
raidhotadd and raidhotremove will only work on a running array. If only 1 out of the 3 drive partitions are functioning within an array, then raidhotadd and raidhotremove are useless.
You can usually tell what drive belongs to what array by looking at your RAID configuration file, which is /etc/raidtab.
(*). Useful command summary:
vi /etc/raidtab dmesg cat /proc/mdstat raidhotadd /dev/md? /dev/hd?? raidhotadd raidhotremove lsraid -a /dev/md0 lsraid -A -d /dev/hdc2
(man). lsraid usage examples
- lsraid -A -a /dev/md0
- Display a short listing of the md0 device.
- lsraid -A -d /dev/sda1
- Display a short listing of the array that sda1 belongs to.
- lsraid -A -f -a /dev/md0
- Display the failed devices belonging to the md0 device.
- lsraid -D -l -a /dev/md0
- Display a long dump of the on-disk md superblock of every disk in
- md0.
- lsraid -D -a /dev/md0 -d /dev/sda1
- Display a short discription of the disks in md0 as well as a short
- description of the disk sda1. sda1 will only be described once if
- it belongs to md0.
- lsraid -R -a /dev/md0 -a /dev/md1 -a /dev/md2
- Display a description of the arrays in an output format suitable
- for using in raidtab(5) files. Note that if md0 and md1 are raid0
- arrays and md2 is a raid1 created from md0 and md1, this command
- will output the information in the correct order.
- lsraid -R -p
- Scan all block devices in /proc/partitions and display all discov-
- ered md devices in a format suitable for using in raidtab(5) files.
Command Reference by Example
vi /etc/raidtab
dmesg
cat /proc/mdstat
raidhotadd /dev/mdN /dev/hdXN
raidhotremove
lsraid -a /dev/mdN
lsraid -A -d /dev/hdcN
raidstop
raidsetfaulty
mkraid -f (dangerous!)
badblocks
mount /proc /proc -t proc