For the purpose of this example, I have enough room on my local hard disk to create four files at 1GB each. I want to create one disk group that contains four disks. The disk group will contain two failure groups and each failure group will be created using two disks.

From within the Solaris operating environment, perform the following actions:

1. Create Files using "dd"

   # mkdir /asmdisks
   # chown oracle:dba /asmdisks
   # su - oracle
   
   $ dd if=/dev/zero of=/asmdisks/_file_disk1 bs=1k count=1000000
   $ dd if=/dev/zero of=/asmdisks/_file_disk2 bs=1k count=1000000
   $ dd if=/dev/zero of=/asmdisks/_file_disk3 bs=1k count=1000000
   $ dd if=/dev/zero of=/asmdisks/_file_disk4 bs=1k count=1000000

   We now have four files to be used as virtual disks of 1GB each:

   $ ls -l /asmdisks
   total 8004032
   -rw-r--r--   1 oracle   dba      1024000000 Jun 10 21:06 _file_disk1
   -rw-r--r--   1 oracle   dba      1024000000 Jun 10 21:08 _file_disk2
   -rw-r--r--   1 oracle   dba      1024000000 Jun 10 21:09 _file_disk3
   -rw-r--r--   1 oracle   dba      1024000000 Jun 10 21:09 _file_disk4

2. Associate Loop Device with the New Files

   The next step is to use the lofiadm command to associate a loop device with a file. The utility lofiadm on Solaris performs the same functionality as the losetup does on Linux. It connects a file (which is presumeably a filesystem image) with a block device, so that it can be mounted, mkfs'd, etc. In our case, we simply want to use it as a RAW device. This needs to be performed as the root user account:

   # lofiadm -a /asmdisks/_file_disk1
   /dev/lofi/1
   
   # lofiadm -a /asmdisks/_file_disk2
   /dev/lofi/2
   
   # lofiadm -a /asmdisks/_file_disk3
   /dev/lofi/3
   
   # lofiadm -a /asmdisks/_file_disk4
   /dev/lofi/4

   Now that the device(s) are created, we should now query the new devices using the following command:

   # ls -lL /dev/lofi/*
   brw-------   1 root   sys    147,  1 Mar 23 01:25 /dev/lofi/1
   brw-------   1 root   sys    147,  2 Jun 10 20:16 /dev/lofi/2
   brw-------   1 root   sys    147,  3 Jun 10 20:16 /dev/lofi/3
   brw-------   1 root   sys    147,  4 Jun 10 20:16 /dev/lofi/4

   We can now use the major and minor device numbers for each of the new physical device files with mknod. Take the new virtual disk device /dev/lofi/1 for example, it has a major number of 147 and minor number of 1. The next step is to use the mknod utility. The mknod utility is used to create both character and block special files. The following creates four new (character) special files for our new virtual devices using the major and minor numbers identified from the listing above:

   # mknod /asmdisks/vdisk1 c 147 1
   # mknod /asmdisks/vdisk2 c 147 2
   # mknod /asmdisks/vdisk3 c 147 3
   # mknod /asmdisks/vdisk4 c 147 4

3. Change Ownership of RAW Devices

   Finally, let's change the ownership of all four RAW devices:

   # chown oracle:dba /asmdisks/vdisk1
   # chown oracle:dba /asmdisks/vdisk2
   # chown oracle:dba /asmdisks/vdisk3
   # chown oracle:dba /asmdisks/vdisk4

After completeting the above steps, the O/S will see four free 'devices' now available for ASM!

For the purpose of this example, I already have an ASM instance running on the same node named "+ASM".

For a detailed article on configuring ASM, see my article: "Manually Creating an ASM Instance".

Let's start by determining if Oracle can find these four new disks:

# ls -l /asmdisks/vdisk[1234]
crw-r--r--   1 oracle   dba      147,  1 Jun 10 21:17 /asmdisks/vdisk1
crw-r--r--   1 oracle   dba      147,  2 Jun 10 21:17 /asmdisks/vdisk2
crw-r--r--   1 oracle   dba      147,  3 Jun 10 21:18 /asmdisks/vdisk3
crw-r--r--   1 oracle   dba      147,  4 Jun 10 21:18 /asmdisks/vdisk4

The following assumes you have the initialization parameter asm_diskstring set properly for the ASM instance. For the purpose of this example, my asm_diskstring parameter is set as follows: *.asm_diskstring='/asmdisks/*'

$ ORACLE_SID=+ASM; export ORACLE_SID
$ sqlplus "/ as sysdba"

SQL> SELECT group_number, disk_number, mount_status, header_status, state, path
  2  FROM   v$asm_disk

GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU STATE    PATH
------------ ----------- ------- ------------ -------- -------------------
           0           0 CLOSED  CANDIDATE    NORMAL   /asmdisks/vdisk1
           0           1 CLOSED  CANDIDATE    NORMAL   /asmdisks/vdisk2
           0           2 CLOSED  CANDIDATE    NORMAL   /asmdisks/vdisk3
           0           3 CLOSED  CANDIDATE    NORMAL   /asmdisks/vdisk4

Using SQL*Plus, the following will create a disk group with normal redundancy and two failure groups:

$ ORACLE_SID=+ASM; export ORACLE_SID

$ sqlplus "/ as sysdba"

SQL> CREATE DISKGROUP testdb_data1 NORMAL REDUNDANCY
  2  FAILGROUP vcontroller1 DISK '/asmdisks/vdisk1', '/asmdisks/vdisk2'
  3  FAILGROUP vcontroller2 DISK '/asmdisks/vdisk3', '/asmdisks/vdisk4';

Diskgroup created.

Now, let's take a look at the new disk group and disk details:

SQL> select group_number, name, total_mb, free_mb, state, type
  2  from v$asm_diskgroup;

GROUP_NUMBER NAME             TOTAL_MB    FREE_MB STATE       TYPE
------------ -------------- ---------- ---------- ----------- ------
           1 TESTDB_DATA1         3904       3798 MOUNTED     NORMAL

SQL> select group_number, disk_number, mount_status, header_status, state, path, failgroup
  2  from v$asm_disk;

GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU STATE    PATH              FAILGROUP
------------ ----------- ------- ------------ -------- ----------------- ------------
           1           0 CACHED  MEMBER       NORMAL   /asmdisks/vdisk1  VCONTROLLER1
           1           1 CACHED  MEMBER       NORMAL   /asmdisks/vdisk2  VCONTROLLER1
           1           2 CACHED  MEMBER       NORMAL   /asmdisks/vdisk3  VCONTROLLER2
           1           3 CACHED  MEMBER       NORMAL   /asmdisks/vdisk4  VCONTROLLER2

...

/usr/sbin/lofiadm -a /asmdisks/_file_disk1; sleep 2
/usr/sbin/lofiadm -a /asmdisks/_file_disk2; sleep 2
/usr/sbin/lofiadm -a /asmdisks/_file_disk3; sleep 2
/usr/sbin/lofiadm -a /asmdisks/_file_disk4; sleep 2

/bin/chown oracle:dba /asmdisks/vdisk1
/bin/chown oracle:dba /asmdisks/vdisk2
/bin/chown oracle:dba /asmdisks/vdisk3
/bin/chown oracle:dba /asmdisks/vdisk4

sleep 120

su - $ORACLE_OWNER -c "$ORACLE_HOME/bin/dbstart"

su - $ORACLE_OWNER -c "lsnrctl start listener"

...

The next step is to edit the /etc/oratab file to allow the dbora script to automatically start and stop databases. Simply alter the final field in the +ASM and TESTDB entry from N to Y. NOTE: Ensure that the ASM instance is started BEFORE any databases that are making use of disk groups contained in it.

...
+ASM:/u01/app/oracle/product/10.1.0/db_1:Y
TESTDB:/u01/app/oracle/product/10.1.0/db_1:Y
... 

The final step to manually edit the script /etc/inittab so that the entry for init.cssd comes before running the runlevel 3. As explained in Metalink Note ID: 264235.1, the fix is as follows:

• Orignal /etc/inittab file:

  (...)
  s2:23:wait:/sbin/rc2  >/dev/msglog 2<>/dev/msglog </dev/console
  s3:3:wait:/sbin/rc3   >/dev/msglog 2<>/dev/msglog </dev/console
  s5:5:wait:/sbin/rc5   >/dev/msglog 2<>/dev/msglog </dev/console
  (...)
  h1:3:respawn:/etc/init.d/init.cssd run >/dev/null 2>&1 </dev/null

• Modified /etc/inittab file:

  (...)
  s2:23:wait:/sbin/rc2  >/dev/msglog 2<>/dev/msglog </dev/console
  h1:3:respawn:/etc/init.d/init.cssd run >/dev/null 2>&1 </dev/null
  s3:3:wait:/sbin/rc3   >/dev/msglog 2<>/dev/msglog </dev/console
  s5:5:wait:/sbin/rc5   >/dev/msglog 2<>/dev/msglog </dev/console
  (...)