For all examples in this document, I will be utilizing a Sun Blade 150 connected to a Sun StorEDGE D1000 Disk Array containing twelve 9.1GB / 10000 RPM / UltraSCSI disk drives for a total disk array capacity of 108GB. The disk array is connected to the Sun Blade 150 using a Dual Differential Ultra/Wide SCSI (X6541A) host adapter. In the Sun StorEDGE D1000 Disk Array, the system identifies the drives as follows:

Controller 1	Controller 2
c1t0d0   -    (d0)	c2t0d0   -    (d0)
c1t1d0   -    (d0)	c2t1d0   -    (d1)
c1t2d0   -    (d1)	c2t2d0   -    (d1)
c1t3d0   -    (d20)	c2t3d0   -    (d20)
c1t4d0   -    (d3)	c2t4d0   -    (d3)
c1t5d0   -    (d3)	c2t5d0   -    (d4)

d0 : RAID 0 - Stripe
d1 : RAID 0 - Concatenation
d20 : RAID 1 - Mirror
d3 : RAID 5
d4 : Hot Spare

From the configuration above, you can see we have plenty of disk drives to utilize for our examples!

The following is the partition tables from one of the twelve hard drives:

format> verify

Primary label contents:

Volume name = <        >
ascii name  = 
pcyl        = 4926
ncyl        = 4924
acyl        =    2
nhead       =   27
nsect       =  133
Part      Tag    Flag     Cylinders        Size            Blocks
0 unassigned    wm       0 -   57      101.70MB    (58/0/0)     208278
1 unassigned    wm       0               0         (0/0/0)           0
2     backup    wm       0 - 4923        8.43GB    (4924/0/0) 17682084
3 unassigned    wm       0               0         (0/0/0)           0
4 unassigned    wm       0               0         (0/0/0)           0
5 unassigned    wm       0               0         (0/0/0)           0
6 unassigned    wm       0               0         (0/0/0)           0
7 unassigned    wm      58 - 4923        8.33GB    (4866/0/0) 17473806

At a bare minimum, DiskSuite requires a minimum of three state database replicas. The system will only stay running with exactly half or more state database replicas available at any one time. The system will panic when less than half the state database replicas are available. The system will not reboot without one more than half the total state database replicas. Instead, it will go into single-user mode for administrative tasks.

State database replicas are created on disk slices using the metadb. Keep in mind that state database replicas can only be created on slices that are not in use. (i.e. have no file system or being used to store RAW data). You cannot create state database replicas on slices on partitions that contain a file system, root (/), /usr, or swap. State database replicas can be created on slices that will be part of a metadevice, but will need to be created BEFORE adding the slice to a metadevice.

In the following example, I will create one state database replica on each of the first 11 disk drives in the D1000 Disk Array using the metadb command. On the twelfth disk, I will give an example of how to create two state database replicas on the same slice. In total I will be creating 13 state database replicas on 12 twelve disks. The replicas will be created on slice 7 for each disk. (This is the slice that we created to be be used as a metadevice for each disk in the disk array.) I will create the state database replicas on the tweleve disks using the following methods:

1. The first four initial state database replicas on the first four disks in the disk array using the -a and -f command line options to the metadb command.
2. Then create seven more replicas just using the -a option to the metadb command.
3. Then use the -c option to the metadb command on the twelfth disk to give an example of how to create two replicas on a single slice.

• The -a switch tells metadb to attach a new database device. The /etc/system file is automatically updated with the new information to tell the system to reattach the devices at boot-time and the /etc/lvm/mddb.cf file is updated. An alternate way to create replicas is by defining them in the /etc/lvm/md.tab file and specifying the assigned name at the command line in the form, mddbnn, where nn is a two-digit number given to the replica definitions. Refer to the md.tab(4) man page for instructions on setting up replicas in that file.
• The -f option is used to create the initial state database. It is also used to force the deletion of replicas below the minimum of one. (The -a and -f options should be used together only when no state databases exist.)

• The -a switch tells metadb to attach a new database device. The /etc/system file is automatically updated with the new information to tell the system to reattach the devices at boot-time and the /etc/lvm/mddb.cf file is updated. An alternate way to create replicas is by defining them in the /etc/lvm/md.tab file and specifying the assigned name at the command line in the form, mddbnn, where nn is a two-digit number given to the replica definitions. Refer to the md.tab(4) man page for instructions on setting up replicas in that file.

• The -a switch tells metadb to attach a new database device. The /etc/system file is automatically updated with the new information to tell the system to reattach the devices at boot-time and the /etc/lvm/mddb.cf file is updated. An alternate way to create replicas is by defining them in the /etc/lvm/md.tab file and specifying the assigned name at the command line in the form, mddbnn, where nn is a two-digit number given to the replica definitions. Refer to the md.tab(4) man page for instructions on setting up replicas in that file.
• The -c switch is used to determine the number of database replicas that will be created on each of the specified slices. In our case, we're creating two replicas on one slice.

• The -d deletes all replicas that are located on the specified slice. The /etc/system file is automatically updated with the new information and the /etc/lvm/mddb.cf file is updated.

Ok, now lets put it back!

# metadb -a c2t4d0s7

• Striped Metadevices
• Concatenated Metadevices
• Concatenated Striped Metadevices

A simple metadevice is called so because they are made only from slices. Simple metadevices can be used directly or as the basic building block for mirrors and trans metadevices.

NOTE: Sometimes a striped metadevice is called a stripe. Other times, stripe refers to the component blocks of a striped concatenation. "To stripe" means to spread I/O requests across disks by chunking parts of the disks and mapping those chunks to a virtual device (a metadevice). Both striping concatenation is also classified as RAID Level 0.

The data in a striped metadevice is arranged across two or more slices. The striping alternates equally-sized segments of data across two or more slices to form one logical storage unit. These segments are interleaved round-robin, so that the combined space is made alternately from each slice. Sort of like a shuffled deck of cards.

1. The following example creates a striped metadevice using 3 slices named /dev/md/rdsk/d0 using the metainit command. Of the twelve disks available in the D1000 Disk Array, I will be using slices c1t0d0s7, c2t0d0s7, c1t1d0s7 as follows:

   # metainit d0 1 3 c1t0d0s7 c2t0d0s7 c1t1d0s7 -i 32k
   d0: Concat/Stripe is setup

2. Use the metastat command to query your new metadevice:

   # metastat d0
   d0: Concat/Stripe
       Size: 52407054 blocks
       Stripe 0: (interlace: 64 blocks)
           Device              Start Block  Dbase
           c1t0d0s7                3591     Yes
           c2t0d0s7                3591     Yes
           c1t1d0s7                3591     Yes

   Let's explain the details of the above example. First notice that the new striped metadevice, d0, consists of a single stripe (Stripe 0) made of three slices (c1t0d0s7, c2t0d0s7, c1t1d0s7). The -i option sets the interlace to 32KB. (The interlace cannot be less than 8KB, nor greater than 100MB.) If interlace were not specified on the command line, the striped metadevice would use the default of 16KB. When using the metastat command to verify our metadevice, we can see from all disks belonging to Stripe 0, that this is a stripped metadevice. Also, that the interlace is 32k (512 * 64 blocks) as we defined it. The total size of the stripe is 26,832,411,648 bytes (512 * 52407054 blocks).

3. Now that we have created our simple metadevice (a stripe), we can now pretend that the metadevice is a big partition (slice) on which we can do the usual file system things. Let's now create a UFS file system using the newfs command. I want to create a UFS file system with an 8KB block size:

   # newfs -i 8192 /dev/md/rdsk/d0
   newfs: construct a new file system /dev/md/rdsk/d0: (y/n)? y
   /dev/md/rdsk/d0:        52407054 sectors in 14594 cylinders of 27 tracks, 133 sectors
           25589.4MB in 913 cyl groups (16 c/g, 28.05MB/g, 3392 i/g)
   super-block backups (for fsck -F ufs -o b=#) at:
    32, 57632, 115232, 172832, 230432, 288032, 345632, 403232, 460832, 518432,
    576032, 633632, 691232, 748832, 806432, 864032, 921632, 979232, 1036832,
    1094432, 1152032, 1209632, 1267232, 1324832, 1382432, 1440032, 1497632,
    1555232, 1612832, 1670432, 1728032, 1785632, 1838624, 1896224, 1953824,
    2011424, 2069024, 2126624, 2184224, 2241824, 2299424, 2357024, 2414624,
   
                     <----------   SNIP   ---------->
   
    50909216, 50966816, 51024416, 51082016, 51139616, 51197216, 51254816,
    51312416, 51370016, 51427616, 51480608, 51538208, 51595808, 51653408,
    51711008, 51768608, 51826208, 51883808, 51941408, 51999008, 52056608,
    52114208, 52171808, 52229408, 52287008, 52344608, 52402208,

4. Finally, we mount the file system on /db0 as follows:

   # mkdir /db0
   # mount -F ufs /dev/md/dsk/d0 /db0

5. To ensure that this new file system is mounted each time the machine is started, insert the following line into you /etc/vfstab file (all on one line with tabs separating the fields):

   /dev/md/dsk/d0       /dev/md/rdsk/d0      /db0  ufs     2       yes     -

A DiskSuite Concatenated Metadevice (often called just a Concatenation) is one of three types of simple metadevices.

• Striped Metadevices
• Concatenated Metadevices
• Concatenated Striped Metadevices

A simple metadevice is called so because they are made only from slices. Simple metadevices can be used directly or as the basic building block for mirrors and trans metadevices.

The data for a concatenated metadevice is organized serially and adjacently across disk slices, forming one logical storage unit. Many system administrators use a concatenated metadevice to get more storage capacity by logically combining the capacities of several slices. It is possible to add more slices to the concatenated metadevice as the demand for storage grows. A concatenated metadevice enables you to dynamically expand storage capacity and file system sizes online! With a concatenated metadevice you can add slices even if the other slices are currently active.

NOTE: You can also create a concatenated metadevice from a single slice. You could, for example, create a single-slice concatenated metadevice. Later, when you need more storage, you can add more slices to the concatenated metadevice.

1. The following example creates a concatenated metadevice using 3 slices named /dev/md/rdsk/d1 using the metainit command. Of the twelve disks available in the D1000 Disk Array, I will be using slices c2t1d0s7, c1t2d0s7, c2t2d0s7 as follows:

   # metainit d1 3 1 c2t1d0s7 1 c1t2d0s7 1 c2t2d0s7
   d1: Concat/Stripe is setup

2. Use the metastat command to query your new (or in our example all) metadevices:

   # metastat
   d0: Concat/Stripe
       Size: 52407054 blocks
       Stripe 0: (interlace: 64 blocks)
           Device              Start Block  Dbase
           c1t0d0s7                3591     Yes
           c2t0d0s7                3591     Yes
           c1t1d0s7                3591     Yes
   
   d1: Concat/Stripe
       Size: 52410645 blocks
       Stripe 0:
           Device              Start Block  Dbase
           c2t1d0s7                3591     Yes
       Stripe 1:
           Device              Start Block  Dbase
           c1t2d0s7                3591     Yes
       Stripe 2:
           Device              Start Block  Dbase
           c2t2d0s7                3591     Yes

   Let's explain the details of the above example. First notice that the new striped metadevice, d1, consists of three stripes (Stripe 0, Stripe 1, Stripe 2,) each made from a single slice (c2t1d0s7, c1t2d0s7, c2t2d0s7 respectively). When using the metastat command to verify our metadevice, we can see this is a concatenation from the fact of having multiple Stripes. The total size of the concatenation is 26,834,250,240 bytes (512 * 52410645 blocks).

3. Now that we have created our simple metadevice (a concatenation), we can now pretend that the metadevice is a big partition (slice) on which we can do the usual file system things. Let's now create a UFS file system using the newfs command. I want to create a UFS file system with an 8KB block size:

   # newfs -i 8192 /dev/md/rdsk/d1
   newfs: construct a new file system /dev/md/rdsk/d1: (y/n)? y
   Warning: 1 sector(s) in last cylinder unallocated
   /dev/md/rdsk/d1:        52410644 sectors in 14595 cylinders of 27 tracks, 133 sectors
           25591.1MB in 913 cyl groups (16 c/g, 28.05MB/g, 3392 i/g)
   super-block backups (for fsck -F ufs -o b=#) at:
    32, 57632, 115232, 172832, 230432, 288032, 345632, 403232, 460832, 518432,
    576032, 633632, 691232, 748832, 806432, 864032, 921632, 979232, 1036832,
    1094432, 1152032, 1209632, 1267232, 1324832, 1382432, 1440032, 1497632,
   
                     <----------   SNIP   ---------->
   
    51312416, 51370016, 51427616, 51480608, 51538208, 51595808, 51653408,
    51711008, 51768608, 51826208, 51883808, 51941408, 51999008, 52056608,
    52114208, 52171808, 52229408, 52287008, 52344608, 52402208,

4. Finally, we mount the file system on /db1 as follows:

   # mkdir /db1
   # mount -F ufs /dev/md/dsk/d1 /db1

5. To ensure that this new file system is mounted each time the machine is started, insert the following line into you /etc/vfstab file (all on one line with tabs separating the fields):

   /dev/md/dsk/d1       /dev/md/rdsk/d1      /db1  ufs     2       yes     -

Before creating a mirror, create the striped metadevices or concatenated metadevices that will make up the mirror.

Any file system including root (/), swap, and /usr, or any application such as a database, can use a mirror.

When creating a mirror, first create a one-way mirror, then attach a second submirror. This starts a resync operation and ensures that data is not corrupted.

To mirror an existing file system, use an additional slice of equal or greater size than the slice already used by the mirror. You can use a concatenated metadevice or striped metadevice of two or more slices that have adequate space to contain the mirror.

You can create a one-way mirror for a future two- or three-way mirror.

You can create up to a three-way mirror. However, two-way mirrors usually provide sufficient data redundancy for most applications, and are less expensive in terms of disk drive costs. A three-way mirror enables you to take a submirror offline and perform a backup while maintaining a two-way mirror for continued data redundancy.

Use the same size slices when creating submirrors. Using different size slices creates unused space in the mirror.

Avoid having slices of submirrors on the same disk. Also, when possible, use disks attached to different controllers to avoid single points-of-failure. For maximum protection and performance, place each submirror on a different physical disk and, when possible, on different disk controllers. For further data availability, use hot spares with mirrors.

Adding additional state database replicas before creating a mirror can increase the mirror's performance. As a general rule, add one additional replica for each mirror you add to the system.

If possible create mirrors from disks consisting of the same disk geometries. The historical reason is that UFS uses disk blocks based on disk geometries. Today, the issue is centered around performance: a mirror composed of disks with different geometries will only be as fast as its slowest disk.

This section will contain the following five examples for creating different types of two-way mirrors:

NOTE: The -f option forces the creation of the first concatenation, d21, which contains the mounted file system root (/) on /dev/dsk/c0t0d0s0. The second concatenation, d22, is created from /dev/dsk/c0t2d0s0. (This slice must be the same size or greater than that of d21) The metainit command with the -m option creates the one-way mirror d20 using the concatenation containing root (/). Next, the metaroot command edits the /etc/vfstab and /etc/system files so that the system may be booted with the root file system (/) on a metadevice. (It is a good idea to run lockfs -fa before rebooting.) After a reboot, the submirror d22 is attached to the mirror, causing a mirror resync. (The system verifies that the concatenations and the mirror are set up, and that submirror d22 is attached.) The ls -l command is run on the root raw device to determine the path to the alternate root device in case the system needs to be booted from it.

The system must contain at least three state database replicas before you can create RAID5 metadevices.

A RAID5 metadevice can only handle a single slice failure.

Follow the 20-percent rule when creating a RAID5 metadevice: because of the complexity of parity calculations, metadevices with greater than about 20 percent writes should probably not be RAID5 metadevices. If data redundancy is needed, consider mirroring.

There are drawbacks to a slice-heavy RAID5 metadevice: the more slices a RAID5 metadevice contains, the longer read and write operations will take if a slice fails.

A RAID5 metadevice must consist of at least three slices.

A RAID5 metadevice can be grown by concatenating additional slices to the metadevice. The new slices do not store parity information, however they are parity protected. The resulting RAID5 metadevice continues to handle a single slice failure.

The interlace value is key to RAID5 performance. It is configurable at the time the metadevice is created; thereafter, the value cannot be modified. The default interlace value is 16 Kbytes. This is reasonable for most applications.

Use the same size disk slices. Creating a RAID5 metadevice from different size slices results in unused disk space in the metadevice.

Do not create a RAID5 metadevice from a slice that contains an existing file system. Doing so will erase the data during the RAID5 initialization process.

RAID5 metadevices cannot be striped, concatenated, or mirrored.

1. The following example creates a RAID 5 metadevice using 3 slices that will be named /dev/md/rdsk/d3 with the metainit command. Of the twelve disks available in the D1000 Disk Array, I will be using slices c1t4d0s7, c2t4d0s7, c1t5d0s7 as follows:

   # metainit d3 -r c1t4d0s7 c2t4d0s7 c1t5d0s7
   d3: RAID is setup

   Let's explain the details of the above example. The RAID5 metadevice d3 is created with the -r option from three slices. Because no interlace is specified, d3 uses the default of 16 Kbytes. The system verifies that the RAID5 metadevice has been set up, and begins initializing the metadevice.

2. Use the metastat command to query your new RAID5 metadevices. After running the above command, the metadevice will go through an initialization state. This may take several minutes to complete. When using the metastat command, you will be able to view how far of the initialization is completed. You must wait for the initialization to finish before you can use the new RAID5 metadevice. The following screenshot shows the RAID5 metadevice during its initialization phase:

   # metastat d3
   d3: RAID
       State: Initializing
       Initialization in progress: 22% done
       Interlace: 32 blocks
       Size: 34936839 blocks
   Original device:
       Size: 34939712 blocks
           Device              Start Block  Dbase State        Hot Spare
           c1t4d0s7                3921     Yes   Initializing
           c2t4d0s7                3921     Yes   Initializing
           c1t5d0s7                3921     Yes   Initializing

   When the disks within the RAID5 metadevice are completed with their initialization phase, this is what it will look like:

   # metastat d3
   d3: RAID
       State: Okay
       Interlace: 32 blocks
       Size: 34936839 blocks
   Original device:
       Size: 34939712 blocks
           Device              Start Block  Dbase State        Hot Spare
           c1t4d0s7                3921     Yes   Okay
           c2t4d0s7                3921     Yes   Okay
           c1t5d0s7                3921     Yes   Okay

3. Now that we have created our RAID5 metadevice, we can now pretend that the metadevice is a big partition (slice) on which we can do the usual file system things. Let's now create a UFS file system using the newfs command. I want to create a UFS file system with an 8KB block size:

   # newfs -i 8192 /dev/md/rdsk/d3
   newfs: construct a new file system /dev/md/rdsk/d3: (y/n)? y
   Warning: 1 sector(s) in last cylinder unallocated
   /dev/md/rdsk/d3:        34936838 sectors in 9729 cylinders of 27 tracks, 133 sectors
           17059.0MB in 609 cyl groups (16 c/g, 28.05MB/g, 3392 i/g)
   super-block backups (for fsck -F ufs -o b=#) at:
    32, 57632, 115232, 172832, 230432, 288032, 345632, 403232, 460832, 518432,
    576032, 633632, 691232, 748832, 806432, 864032, 921632, 979232, 1036832,
    1094432, 1152032, 1209632, 1267232, 1324832, 1382432, 1440032, 1497632,
   
                     <----------   SNIP   ---------->
   
    34016288, 34073888, 34131488, 34189088, 34246688, 34304288, 34361888,
    34419488, 34477088, 34534688, 34592288, 34649888, 34707488, 34765088,
    34822688, 34880288, 34933280,

4. Finally, we mount the file system on /db3 as follows:

   # mkdir /db3
   # mount -F ufs /dev/md/dsk/d3 /db3

5. To ensure that this new file system is mounted each time the machine is started, insert the following line into you /etc/vfstab file (all on one line with tabs separating the fields):

   /dev/md/dsk/d3       /dev/md/rdsk/d3      /db3  ufs     2       yes     -

The trans metadevice normally has two devices: the master device and the logging device. The master contains the file system that is being logged. The logging device contains the log and can be shared by several file systems. It is a sequence of records, each of which describes a change to a file system. Both the master device and the logging device can be a slice or a metadevice.

Though logs can be shared between file systems, heavily-used file systems should have their own logging device.

Small file systems with mostly read operations probably do not need to be logged.

Any UFS, except root (/), can be logged.

Even if you don't have an available slice for a logging device, you can still set up a trans metadevice without a logging device. This is useful if you plan to enable logging on exported file systems, but do not have an available slice for the logging device at this time.

Before creating trans metadevices, identify the slices or metadevice to be used as the master devices and logging devices.

Avoid placing logs on heavily-used disks.

Do not use a RAID5 metadevice as a logging device. Instead, use a mirror for data redundancy.

Logs (the logging device) can be placed on a slice that already contains a state database replica.

Plan on using one megabyte of log space as a minimum, and an additional one megabyte of log space per 100 megabytes of file system data, up to a maximum log size of 64 Mbytes. Logs greater than 64 Mbytes waste space.

The master devices and logging devices of the same trans metadevice should be located on separate drives and possibly separate controllers.

CAUTION: Mirroring logging devices is strongly recommended. Losing the data in a logging device because of device errors can leave a file system in an inconsistent state which fsck(1M) may not be able to fix without user intervention. Using a mirror for the master device is a good idea to ensure data redundancy.

This section will contain the following four examples for creating different types of trans metadevices:

The following example creates a trans metadevice that cannot be unmounted during normal system operation. The UFS file system (the master device) that will be logged is /usr. The /usr file system is currently mounted on /dev/dsk/c0t0d0s6. The logging device will be created on slice /dev/dsk/c1t1d0s0.

1. First, create the trans metadevice with the metainit command. Slice /dev/dsk/c0t0d0s6 contains the /usr file system. The slice to contain the logging device is /dev/dsk/c1t1d0s0. Because /usr cannot be unmounted, the metainit command is run with the -f option to force the creation of the trans device, d6.

   # metainit -f d6 -t /dev/dsk/c0t0d0s6 c1t1d0s0
   d6: Trans is setup

2. Edit the /etc/vfstab file so that the /usr file system now refers to the newly created trans metadevice. In the example snippet, I commented out the original entry for the /usr file system (c0t0d0s6) and added a new entry that refers to the newly created trans metadevice to be mounted to /usr:

   # /dev/dsk/c0t0d0s6     /dev/rdsk/c0t0d0s6      /usr    ufs     1       no      -
   /dev/md/dsk/d6   /dev/md/rdsk/d6   /usr    ufs     1       no      -

3. Reboot the system

   # reboot

4. Logging becomes effective for the file system when the system is rebooted!

5. Finally, lets check the status of the new Trans Metadevice:

   # metastat d6
   d6: Trans
       State: Okay
       Size: 16781040 blocks
       Master Device: c0t0d0s6
       Logging Device: c1t1d0s0
   
           Master Device       Start Block  Dbase
           c0t0d0s6                   0     No
   
   c1t1d0s0: Logging device for d6
       State: Okay
       Size: 202637 blocks
   
           Logging Device      Start Block  Dbase
           c1t1d0s0                5641     No

You can (and should!!!) increase data availability of a trans metadevice by using mirrors for the master AND logging devices. Failure to mirror the logging device could result in significant data loss if the logging slice experiences errors. If you are mirroring the logging device, it is a good idea that the master device be a mirror also.

1. This is an example of how to create a Trans Metadevice where both the master device and the logging device is a mirror.

2. To start off with, we already have a mirrored metadevice named d20 which is mounted on /db20. This will be the file system we want to have being logged. (Not that it matters for this example, but the mirrored metadevice is a two-way mirror that contains two submirrors: d21 and d22.)

3. Next, make a mirror (lets name this one d30) to be used as the logging device. It will be a two-way mirror containing submirrors d31 and d32.

4. First, unmount the file system you want to have logged:

   # umount /db20

5. Next, use the metainit command with the -t option to create the Trans Metadevice, d40:

   # metainit d40 -t d20 d30
   d40: Trans is setup

6. Edit the /etc/vfstab file so that the /db20 file system now refers to the newly created trans metadevice. In the example snippet, I commented out the original entry for the /db20 file system (/dev/md/dsk/d0) and added a new entry that refers to the newly created trans metadevice to be mounted to /db20:

   # /dev/md/dsk/d0   /dev/md/rdsk/d0   /db0    ufs     2       yes     -
   /dev/md/dsk/d40   /dev/md/rdsk/d40   /db0    ufs     2       yes      -

7. Logging becomes effective for the file system when the file system is remounted!

8. On subsequent file system remounts or system reboots, instead of checking the file system, fsck displays a logging message for the metadevice:

   # reboot
   ...
   /dev/md/rdsk/d40: is logging