For all examples in this document, I will be utilizing a Sun Blade 150 connected to a Sun StorEDGE D1000 Disk Array containing twelve 9.1GB / 10000 RPM / UltraSCSI disk drives for a total disk array capacity of 108GB. The disk array is connected to the Sun Blade 150 using a Dual Differential Ultra/Wide SCSI (X6541A) host adapter. In the Sun StorEDGE D1000 Disk Array, the system identifies the drives as follows:

Controller 1	Controller 2
c1t0d0   -    (d0)	c2t0d0   -    (d0)
c1t1d0   -    (d0)	c2t1d0   -    (d1)
c1t2d0   -    (d1)	c2t2d0   -    (d1)
c1t3d0   -    (d20)	c2t3d0   -    (d20)
c1t4d0   -    (d3)	c2t4d0   -    (d3)
c1t5d0   -    (d3)	c2t5d0   -    (d4)

d0 : RAID 0 - Stripe
d1 : RAID 0 - Concatenation
d20 : RAID 1 - Mirror
d3 : RAID 5
d4 : Hot Spare

From the configuration above, you can see we have plenty of disk drives to utilize for our examples! For the examples in this article, I will only be using several of the disks within the D1000 array - in many cases, just enough to demonstrate the use of the Volume Manager commands and component configuration.

The following is the partition tables from one of the twelve hard drives:

format> verify

Primary label contents:

Volume name = <        >
ascii name  = <SUN9.0G cyl 4924 alt 2 hd 27 sec 133>
pcyl        = 4926
ncyl        = 4924
acyl        =    2
nhead       =   27
nsect       =  133
Part      Tag    Flag     Cylinders        Size            Blocks
  0 unassigned    wm       0               0         (0/0/0)           0
  1 unassigned    wm       0               0         (0/0/0)           0
  2     backup    wm       0 - 4923        8.43GB    (4924/0/0) 17682084
  3 unassigned    wm       0               0         (0/0/0)           0
  4 unassigned    wm       0               0         (0/0/0)           0
  5 unassigned    wm       0               0         (0/0/0)           0
  6 unassigned    wm       0               0         (0/0/0)           0
  7        usr    wm       0 - 4922        8.43GB    (4923/0/0) 17678493

At a bare minimum, Volume Manager requires a minimum of three state database replicas. If your system looses a state database replica, Volume Manager will attempt to determine which state database replcas are still valid. Before any of the state database replicas can be considered valid, Volume Manager requires that a majority (half + 1) of the state database replicas be available and in agreement before any of them are considered valid. Solaris Volume Manager calls this algorithm a majority consensus algorithm. The system will not reboot without one more than half the total state database replicas. Instead, it will go into single-user mode for administrative tasks.

State database replicas are created on disk slices using the metadb command. Keep in mind that state database replicas can only be created on slices that are not in use. (i.e. have no file system or being used to store RAW data). You cannot create state database replicas on slices on partitions that contain a file system, root (/), /usr, or swap. State database replicas can be created on slices that will be part of a volume, but will need to be created BEFORE adding the slice to a volume.

In the following example, I will create one state database replica on each of the first 11 disk drives in the D1000 Disk Array using the metadb command. On the twelfth disk, I will give an example of how to create two state database replicas on the same slice. In total I will be creating 13 state database replicas on 12 twelve disks. The replicas will be created on slice 7 for each disk. (This is the slice that we created to be be used for each disk in the disk array.) I will create the 13 state database replicas on the tweleve disks using the following methods:

1. The first four initial state database replicas on the first four disks in the disk array using the -a and -f command line options to the metadb command.
2. Then create seven more replicas just using the -a option to the metadb command.
3. Then use the -c option to the metadb command on the twelfth disk to give an example of how to create two replicas on a single slice.

• The -a switch tells metadb to attach a new database device. The /etc/lvm/mddb.cf file is automatically updated with the new information to tell the system to reattach the devices at boot-time. An alternate way to create replicas in DiskSuite 4.2.1 was by defining them in the /etc/lvm/md.tab file and specifying the assigned name at the command line in the form, mddbnn, where nn is a two-digit number given to the replica definitions. I do not believe this file is used in Solaris 9 Volume Manager. Refer to the md.tab(4) man page for instructions on setting up replicas in that file.
• The -f option is used to create the initial state database. It is also used to force the deletion of replicas below the minimum of one. (The -a and -f options should be used together only when no state databases exist.)

• The -a switch tells metadb to attach a new database device. The /etc/lvm/mddb.cf file is automatically updated with the new information to tell the system to reattach the devices at boot-time. An alternate way in DiskSuite 4.2.1 to create replicas was by defining them in the /etc/lvm/md.tab file and specifying the assigned name at the command line in the form, mddbnn, where nn is a two-digit number given to the replica definitions. I do not believe this file is used in Solaris 9 Volume Manager. Refer to the md.tab(4) man page for instructions on setting up replicas in that file.

• The -a switch tells metadb to attach a new database device. The /etc/lvm/mddb.cf file is automatically updated with the new information to tell the system to reattach the devices at boot-time. An alternate way in DiskSuite 4.2.1 to create replicas was by defining them in the /etc/lvm/md.tab file and specifying the assigned name at the command line in the form, mddbnn, where nn is a two-digit number given to the replica definitions. I do not believe this file is used in Solaris 9 Volume Manager. Refer to the md.tab(4) man page for instructions on setting up replicas in that file.
• The -c switch is used to determine the number of database replicas that will be created on each of the specified slices. In our case, we're creating two replicas on one slice.

• The -d deletes all replicas that are located on the specified slice. The /etc/system file is automatically updated with the new information and the /etc/lvm/mddb.cf file is updated.

Ok, now lets put it back!

# metadb -a c2t4d0s7

• Striped Volumes - (or stripes)
• Concatenated Volumes - (or concatenations)
• Concatenated Striped Volumes - (or contatenated stripes)

These components are made from slices. Simple volumes can be used directly or as the basic building block for mirrors.

NOTE: Sometimes a striped volume is called a stripe. Other times, stripe refers to the component blocks of a striped concatenation. To "stripe" means to spread I/O requests across disks by chunking parts of the disks and mapping those chunks to a virtual device (a volume). Both striping and concatenation are classified as RAID Level 0.

The data in a striped volume is arranged across two or more slices. The striping alternates equally-sized segments of data across two or more slices to form one logical storage unit. These segments are interleaved round-robin, so that the combined space is made alternately from each slice. Sort of like a shuffled deck of cards.

1. The following example creates a striped volume using 3 slices named /dev/md/rdsk/d0 using the metainit command. Of the twelve disks available in the D1000 Disk Array, I will be using slices c1t0d0s7, c2t0d0s7, c1t1d0s7 as follows:

   # metainit d0 1 3 c1t0d0s7 c2t0d0s7 c1t1d0s7 -i 32k
   d0: Concat/Stripe is setup

2. Use the metastat command to query your new volume:

   # metastat d0
   d0: Concat/Stripe
       Size: 52999569 blocks (25 GB)
       Stripe 0: (interlace: 64 blocks)
           Device     Start Block  Dbase   Reloc
           c1t0d0s7      10773     Yes     Yes
           c2t0d0s7      10773     Yes     Yes
           c1t1d0s7      10773     Yes     Yes
   
   Device Relocation Information:
   Device   Reloc  Device ID
   c1t0d0   Yes    id1,sd@SSEAGATE_ST39102LCSUN9.0GLJR76697000019460DB4
   c2t0d0   Yes    id1,sd@SSEAGATE_ST39102LCSUN9.0GLV00222700001005J6Q7
   c1t1d0   Yes    id1,sd@SSEAGATE_ST39102LCSUN9.0GLJR58209000019461YK2

   Let's explain the details of the above example. First notice that the new striped volume, d0, consists of a single stripe (Stripe 0) made of three slices (c1t0d0s7, c2t0d0s7, c1t1d0s7). The -i option sets the interlace to 32KB. (The interlace cannot be less than 8KB, nor greater than 100MB.) If interlace were not specified on the command line, the striped volume would use the default of 16KB. When using the metastat command to verify our volume, we can see from all disks belonging to Stripe 0, that this is a stripped volume. Also, that the interlace is 32k (512 * 64 blocks) as we defined it. The total size of the stripe is 27,135,779,328 bytes (512 * 52999569 blocks).

3. Now that we have created our simple volume (a RAID 0 stripe), we can now pretend that the volume is a big partition (slice) on which we can do the usual file system things. Let's now create a UFS file system using the newfs command. I want to create a UFS file system with an 8KB block size:

   # newfs -i 8192 /dev/md/rdsk/d0
   newfs: /dev/md/rdsk/d0 last mounted as /db0
   newfs: construct a new file system /dev/md/rdsk/d0: (y/n)? y
   Warning: 1 sector(s) in last cylinder unallocated
   /dev/md/rdsk/d0:        52999568 sectors in 14759 cylinders of 27 tracks, 133 sectors
           25878.7MB in 923 cyl groups (16 c/g, 28.05MB/g, 3392 i/g)
   super-block backups (for fsck -F ufs -o b=#) at:
    32, 57632, 115232, 172832, 230432, 288032, 345632, 403232, 460832, 518432,
   Initializing cylinder groups:
   ..................
   super-block backups for last 10 cylinder groups at:
    52459808, 52517408, 52575008, 52632608, 52690208, 52747808, 52805408,
    52863008, 52920608, 52978208,

4. Finally, we mount the file system on /db0 as follows:

   # mkdir /db0
   # mount -F ufs /dev/md/dsk/d0 /db0

5. To ensure that this new file system is mounted each time the machine is started, insert the following line into you /etc/vfstab file (all on one line with tabs separating the fields):

   /dev/md/dsk/d0       /dev/md/rdsk/d0      /db0  ufs     2       yes     -

A Solaris 9 Volume Manager Concatenated Volume (often called just a Concatenation) is one of three types of simple volumes.

• Striped Volumes - (or stripes)
• Concatenated Volumes - (or concatenations)
• Concatenated Striped Volumes - (or contatenated stripes)

These components are made from slices. Simple volumes can be used directly or as the basic building block for mirrors.

The data for a concatenated volume is organized serially and adjacently across disk slices, forming one logical storage unit. Many system administrators use a concatenated volume to get more storage capacity by logically combining the capacities of several slices. It is possible to add more slices to the concatenated volume as the demand for storage grows. A concatenated volume enables you to dynamically expand storage capacity and file system sizes online! With a concatenated volume you can add slices even if the other slices are currently active.

NOTE: You can also create a concatenated volume from a single slice. You could, for example, create a single-slice concatenated volume. Later, when you need more storage, you can add more slices to the concatenated volume.

1. The following example creates a concatenated volume using 3 slices named /dev/md/rdsk/d1 using the metainit command. Of the twelve disks available in the D1000 Disk Array, I will be using slices c2t1d0s7, c1t2d0s7, c2t2d0s7 as follows:

   # metainit d1 3 1 c2t1d0s7 1 c1t2d0s7 1 c2t2d0s7
   d1: Concat/Stripe is setup

2. Use the metastat command to query your new (or in our example all) volumes:

   # metastat
   d1: Concat/Stripe
       Size: 53003160 blocks (25 GB)
       Stripe 0:
           Device     Start Block  Dbase   Reloc
           c2t1d0s7      10773     Yes     Yes
       Stripe 1:
           Device     Start Block  Dbase   Reloc
           c1t2d0s7      10773     Yes     Yes
       Stripe 2:
           Device     Start Block  Dbase   Reloc
           c2t2d0s7      10773     Yes     Yes
   
   d0: Concat/Stripe
       Size: 52999569 blocks (25 GB)
       Stripe 0: (interlace: 64 blocks)
           Device     Start Block  Dbase   Reloc
           c1t0d0s7      10773     Yes     Yes
           c2t0d0s7      10773     Yes     Yes
           c1t1d0s7      10773     Yes     Yes
   
   Device Relocation Information:
   Device   Reloc  Device ID
   c2t1d0   Yes    id1,sd@SSEAGATE_ST39102LCSUN9.0GLJP46564000019451VGF
   c1t2d0   Yes    id1,sd@SSEAGATE_ST39102LCSUN9.0GLJU8183300002007J3Z2
   c2t2d0   Yes    id1,sd@SSEAGATE_ST39102LCSUN9.0GLJM7285500001943H5XD
   c1t0d0   Yes    id1,sd@SSEAGATE_ST39102LCSUN9.0GLJR76697000019460DB4
   c2t0d0   Yes    id1,sd@SSEAGATE_ST39102LCSUN9.0GLV00222700001005J6Q7
   c1t1d0   Yes    id1,sd@SSEAGATE_ST39102LCSUN9.0GLJR58209000019461YK2

   Let's explain the details of the above example. First notice that the new concatenated volume, d1, consists of three stripes (Stripe 0, Stripe 1, Stripe 2,) each made from a single slice (c2t1d0s7, c1t2d0s7, c2t2d0s7 respectively). When using the metastat command to verify our volumes, we can see this is a concatenation from the fact of having multiple Stripes. The total size of the concatenation is 27,137,617,920 bytes (512 * 53003160 blocks).

3. Now that we have created our simple volume (a concatenation), we can now pretend that the volume is a big partition (concatenation) on which we can do the usual file system things. Let's now create a UFS file system using the newfs command. I want to create a UFS file system with an 8KB block size:

   # newfs -i 8192 /dev/md/rdsk/d1
   newfs: construct a new file system /dev/md/rdsk/d1: (y/n)? y
   /dev/md/rdsk/d1:        53003160 sectors in 14760 cylinders of 27 tracks, 133 sectors
           25880.4MB in 923 cyl groups (16 c/g, 28.05MB/g, 3392 i/g)
   super-block backups (for fsck -F ufs -o b=#) at:
    32, 57632, 115232, 172832, 230432, 288032, 345632, 403232, 460832, 518432,
   Initializing cylinder groups:
   ..................
   super-block backups for last 10 cylinder groups at:
    52459808, 52517408, 52575008, 52632608, 52690208, 52747808, 52805408,
    52863008, 52920608, 52978208,

4. Finally, we mount the file system on /db1 as follows:

   # mkdir /db1
   # mount -F ufs /dev/md/dsk/d1 /db1

5. To ensure that this new file system is mounted each time the machine is started, insert the following line into you /etc/vfstab file (all on one line with tabs separating the fields):

   /dev/md/dsk/d1       /dev/md/rdsk/d1      /db1  ufs     2       yes     -

Before creating a mirror, create the striped or concatenated volumes that will make up the mirror.

Any file system including root (/), swap, and /usr, or any application such as a database, can use a mirror. Basically, you can mirror any file system, including existing file systems. You can also mirror large applications, such as the data files for a database.

When creating a mirror, first create a one-way mirror, then attach a second submirror. This starts a resync operation and ensures that data is not corrupted.

To mirror an existing file system, use an additional slice of equal or greater size than the slice already used by the mirror. You can use a concatenated volume or striped volume of two or more slices that have adequate space to contain the mirror.

You can create a one-way mirror for a future two- or three-way mirror.

You can create up to a three-way mirror. However, two-way mirrors usually provide sufficient data redundancy for most applications, and are less expensive in terms of disk drive costs. A three-way mirror enables you to take a submirror offline and perform a backup while maintaining a two-way mirror for continued data redundancy. While any submirror is offline, all reading and writing to the submirror is stopped. This enables system administrators to take backups of other system administration responsibilities. Remember, the submirror is in a read-only state. While the submirror is offline, Solaris Volume Manager is keeping track of all writes to the mirror. When the submirror is brought back online, only portions of the mirror that were written while the submirror was offline are resynchronized.

Use the same size slices when creating submirrors. Using different size slices creates unused space in the mirror.

Avoid having slices of submirrors on the same disk. Also, when possible, use disks attached to different controllers to avoid single points-of-failure. For maximum protection and performance, place each submirror on a different physical disk and, when possible, on different disk controllers. For further data availability, use hot spares with mirrors.

In some cases, mirroring can improve read performance. Write performance, however, will always degrade. If an application is multi-threaded or can take advantage of asynchronous I/O, you will see performance gains. If an application is only single-threaded reading from the volume, you will see no performance gains.

Adding additional state database replicas before creating a mirror can increase the mirror's performance. As a general rule, add one additional replica for each mirror you add to the system.

If possible create mirrors from disks consisting of the same disk geometries. The historical reason is that UFS uses disk blocks based on disk geometries. Today, the issue is centered around performance: a mirror composed of disks with different geometries will only be as fast as its slowest disk.

This section will contain the following five examples for creating different types of two-way mirrors:

NOTE: The -f option forces the creation of the first concatenation, d21, which contains the mounted file system root (/) on /dev/dsk/c0t0d0s0. The second concatenation, d22, is created from /dev/dsk/c0t2d0s0. (This slice must be the same size or greater than that of d21) The metainit command with the -m option creates the one-way mirror d20 using the concatenation containing root (/). Next, the metaroot command edits the /etc/vfstab and /etc/system files so that the system may be booted with the root file system (/) on a volume. (It is a good idea to run lockfs -fa before rebooting.) After a reboot, the submirror d22 is attached to the mirror, causing a mirror resync. (The system verifies that the concatenations and the mirror are set up, and that submirror d22 is attached.) The ls -l command is run on the root raw device to determine the path to the alternate root device in case the system needs to be booted from it.

The system must contain at least three state database replicas before you can create RAID5 volumes.

A RAID5 volume can only handle a single slice failure.

Follow the 20-percent rule when creating a RAID5 volume: because of the complexity of parity calculations, volumes with greater than about 20 percent writes should probably not be RAID5 volumes. If data redundancy is needed, consider mirroring.

There are drawbacks to a slice-heavy RAID5 volume: the more slices a RAID5 volume contains, the longer read and write operations will take if a slice fails.

A RAID5 volume must consist of at least three slices.

A RAID5 volume can be grown by concatenating additional slices to the volume. The new slices do not store parity information, however they are parity protected. The resulting RAID5 volume continues to handle a single slice failure.

The interlace value is key to RAID5 performance. It is configurable at the time the volume is created; thereafter, the value cannot be modified. The default interlace value is 16 Kbytes. This is reasonable for most applications.

Use the same size disk slices. Creating a RAID5 volume from different size slices results in unused disk space in the volume.

Do not create a RAID5 volume from a slice that contains an existing file system. Doing so will erase the data during the RAID5 initialization process.

RAID5 volumes cannot be striped, concatenated, or mirrored.

1. The following example creates a RAID 5 volume using 3 slices that will be named /dev/md/rdsk/d3 with the metainit command. Of the twelve disks available in the D1000 Disk Array, I will be using slices c1t4d0s7, c2t4d0s7, and c1t5d0s7 as follows:

   # metainit d3 -r c1t4d0s7 c2t4d0s7 c1t5d0s7
   d3: RAID is setup

   Let's explain the details of the above example. The RAID5 volume d3 is created with the -r option from three slices. Because no interlace is specified, d3 uses the default of 16 Kbytes. The system verifies that the RAID5 volume has been set up, and begins initializing the volume.

2. Use the metastat command to query your new RAID5 volumes. After running the above command, the volume will go through an initialization state. This may take several minutes to complete. When using the metastat command, you will be able to view how far of the initialization is completed. You must wait for the initialization to finish before you can use the new RAID5 volume. The following screenshot shows the RAID5 volume during its initialization phase:

   # metastat d3
   d3: RAID
       State: Initializing
       Initialization in progress: 32.0% done
       Interlace: 32 blocks
       Size: 35331849 blocks (16 GB)
   Original device:
       Size: 35334720 blocks (16 GB)
           Device     Start Block  Dbase        State Reloc  Hot Spare
           c1t4d0s7      11103       Yes Initializing   Yes
           c2t4d0s7      11103       Yes Initializing   Yes
           c1t5d0s7      11103       Yes Initializing   Yes
   
   Device Relocation Information:
   Device   Reloc  Device ID
   c1t4d0   Yes    id1,sd@SSEAGATE_ST39102LCSUN9.0GLJP248260000194511NU
   c2t4d0   Yes    id1,sd@SSEAGATE_ST39102LCSUN9.0GLJP1841500002945H5FE
   c1t5d0   Yes    id1,sd@SSEAGATE_ST39102LCSUN9.0GLJE34597000029290C8N

   When the disks within the RAID5 volume are completed with their initialization phase, this is what it will look like:

   # metastat d3
   d3: RAID
       State: Okay
       Interlace: 32 blocks
       Size: 35331849 blocks (16 GB)
   Original device:
       Size: 35334720 blocks (16 GB)
           Device     Start Block  Dbase        State Reloc  Hot Spare
           c1t4d0s7      11103       Yes         Okay   Yes
           c2t4d0s7      11103       Yes         Okay   Yes
           c1t5d0s7      11103       Yes         Okay   Yes
   
   Device Relocation Information:
   Device   Reloc  Device ID
   c1t4d0   Yes    id1,sd@SSEAGATE_ST39102LCSUN9.0GLJP248260000194511NU
   c2t4d0   Yes    id1,sd@SSEAGATE_ST39102LCSUN9.0GLJP1841500002945H5FE
   c1t5d0   Yes    id1,sd@SSEAGATE_ST39102LCSUN9.0GLJE34597000029290C8N

3. Now that we have created our RAID5 volume, we can now pretend that the volume is a big partition (slice) on which we can do the usual file system things. Let's now create a UFS file system using the newfs command. I want to create a UFS file system with an 8KB block size:

   # newfs -i 8192 /dev/md/rdsk/d3
   newfs: construct a new file system /dev/md/rdsk/d3: (y/n)? y
   Warning: 1 sector(s) in last cylinder unallocated
   /dev/md/rdsk/d3:        35331848 sectors in 9839 cylinders of 27 tracks, 133 sectors
           17251.9MB in 615 cyl groups (16 c/g, 28.05MB/g, 3392 i/g)
   super-block backups (for fsck -F ufs -o b=#) at:
    32, 57632, 115232, 172832, 230432, 288032, 345632, 403232, 460832, 518432,
   Initializing cylinder groups:
   ............
   super-block backups for last 10 cylinder groups at:
    34765088, 34822688, 34880288, 34933280, 34990880, 35048480, 35106080,
    35163680, 35221280, 35278880,

4. Finally, we mount the file system on /db3 as follows:

   # mkdir /db3
   # mount -F ufs /dev/md/dsk/d3 /db3

5. To ensure that this new file system is mounted each time the machine is started, insert the following line into you /etc/vfstab file (all on one line with tabs separating the fields):

   /dev/md/dsk/d3       /dev/md/rdsk/d3      /db3  ufs     2       yes     -