To take inventory of your database

1. Obtain the list of the database application components.

   [oragrid@db1 ~] crs_stat -t
   Name           Type           Target    State     Host
   ------------------------------------------------------------
   ora....D1.inst application    ONLINE    ONLINE    punl...5001
   ora....D2.inst application    ONLINE    ONLINE    punl...5002
   ora.PROD.db    application    ONLINE    ONLINE    punl...5002
   ora....SM1.asm application    ONLINE    ONLINE    punl...5001
   ora....01.lsnr application    ONLINE    ONLINE    punl...5001
   ora....001.gsd application    ONLINE    ONLINE    punl...5001
   ora....001.ons application    ONLINE    ONLINE    punl...5001
   ora....001.vip application    ONLINE    ONLINE    punl...5001
   ora....SM2.asm application    ONLINE    ONLINE    punl...5002
   ora....02.lsnr application    ONLINE    ONLINE    punl...5002
   ora....002.gsd application    ONLINE    ONLINE    punl...5002
   ora....002.ons application    ONLINE    ONLINE    punl...5002
   ora....002.vip application    ONLINE    ONLINE    punl...5002
   
   [oragrid@db1 ~] srvctl status database -d PROD
   Instance PROD1 is running on node punlxy5001
   Instance PROD2 is running on node punlxy5002
   
   

2. Obtain the list of raw partitions and the ASM disk group information.

   SQL> select name, state,TOTAL_MB, 
   FREE_MB, REQUIRED_MIRROR_FREE_MB, USABLE_FILE_MB  from v$asm_diskgroup;
   
   NAME      STATE         TOTAL_MB    FREE_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB
   --------------------- ---------- ---------- ----------------------- --------------
   DATA      MOUNTED          20444      20349                       0          20349
   FRA       MOUNTED          20444      20349                       0          20349
   
   SQL> select name, state, header_status, path,TOTAL_MB, 
   FREE_MB from v$asm_disk;
   NAME       STATE    HEADER_STATU PATH                               TOTAL_MB   FREE_MB
   ------------------- ------------ ---------------------------------- --------- --------
   DATA_0000  NORMAL   MEMBER       /dev/rdsk/c1t50060E800564F774d40s6 10222     10173
   DATA_0001  NORMAL   MEMBER       /dev/rdsk/c1t50060E800564F774d37s6 10222     10176
   FRA_0000   NORMAL   MEMBER       /dev/rdsk/c1t50060E800564F774d30s6 10222     10175
   FRA_0001   NORMAL   MEMBER       /dev/rdsk/c1t50060E800564F774d29s6 10222     10174
   
   4 rows selected.

To convert raw Solaris partitions in-place into VxVM raw volumes

1. Use prtvtoc to collect the partition details. Make a note of starting sector of slice and slice length of each disk.

   Store the prtvtoc information for all the ASM used disks before starting conversion. This information may be needed in case you want to roll back from VxVM to ASM.

     # prtvtoc /dev/rdsk/c3t50060E800564F774d40s6 
   * /dev/rdsk/c3t50060E800564F774d40s6 partition map 
   * Dimensions: 
   *     512 bytes/sector 
   *     512 sectors/track 
   *      15 tracks/cylinder 
   *    7680 sectors/cylinder 
   *    2730 cylinders 
   *    2728 accessible cylinders 
   * Flags: 
   *   1: unmountable 
   *  10: read-only 
   * 
   *  	     		
                         First		Sector	   Last 
   * Partition	Tag	Flags	Sector	Count		   Sector   Mount Directory   
          2	    5	   01	   0		  20951040	 20951039   
          6	    0	   00	  15360	20935680 	20951039

   In this case, the first sector is 15360 and the sector length is 20935680.

2. Use the vxdisk command to get the disk device name and other information.

   # /opt/VRTS/bin/vxdisk list c3t50060E800564F774d40
   Device:    hitachi_usp-vm0_031c
   devicetag: hitachi_usp-vm0_031c
   type:      auto
   info:      format=none
   flags:     online ready private autoconfig invalid
   pubpaths:  block=/dev/vx/dmp/ hitachi_usp-vm0_031c 
              s2 char=/dev/vx/rdmp/ hitachi_usp-vm0_031c s2
   guid:      -
   udid:      HITACHI%5FOPEN-V%20%20%20%20%20%20-SUN%5F064F7%5F0301
   site:      -
   Multipathing information:
   numpaths:   1
   c3t50060E800564F774d40s2        state=enabled
   

   The following is the information about all the disks used for the ASM database.

   Disk	Start Slice	Length	Device Name
   c3t50060E800564F774d40s6	15360	20935680	hitachi_usp-vm0_031c
   c3t50060E800564F774d37s6	15360	20935680	hitachi_usp-vm0_0319
   c3t50060E800564F774d30s6	15360	20935680	hitachi_usp-vm0_0312
   c3t50060E800564F774d29s6	15360	20935680	hitachi_usp-vm0_0311

3. Shut down the Oracle RAC database and unmount all ASM disk groups used by this database from all the nodes of a cluster.

   $ srvctl stop database -d PROD

   Log in to the ASM instance, and dismount the disk groups.

   $ export ORACLE_SID=+ASM
   $ sqlplus "/as sysasm" 
   SQL> alter diskgroup data dismount; 
   Diskgroup altered. 
   
   SQL> alter diskgroup fra dismount; 
   Diskgroup altered.

4. Use the vxdisksetup utility to convert all the ASM raw partitions into VxVM disks.

   # /opt/VRTS/bin/vxdisksetup -i hitachi_usp-vm0_031c puboffset=15360 \
   publen=20935680 privoffset=0 privlen=8192 format=sliced 
   # /opt/VRTS/bin/vxdisksetup -i hitachi_usp-vm0_0319 puboffset=15360 \
   publen=20935680 privoffset=0 privlen=8192 format=sliced 
   # /opt/VRTS/bin/vxdisksetup -i hitachi_usp-vm0_0312 puboffset=15360 \
   publen=20935680 privoffset=0 privlen=8192 format=sliced 
   # /opt/VRTS/bin/vxdisksetup -i hitachi_usp-vm0_0311 puboffset=15360 \
   publen=20935680 privoffset=0 privlen=8192 format=sliced

   Ensure that puboffset is same as the start sector and publen is same as the slice length of the ASM raw partitions. privlen can be set to 8192 that is used to store VxVM disk metadata. You can use format=sliced to get the two regions on separate slices.

   Note:

   If this operation fails, perform the rollback steps to revert to the original setup.

5. Verify VTOC of the disk to ensure that there are two slices for the private and public regions.

   # prtvtoc /dev/rdsk/c3t50060E800564F774d40s6
   * /dev/rdsk/c3t50060E800564F774d40s6 partition map
   *
   * Dimensions:
   *     512 bytes/sector
   *     512 sectors/track
   *      15 tracks/cylinder
   *    7680 sectors/cylinder
   *    2730 cylinders
   *    2728 accessible cylinders
   *
   * Flags:
   *   1: unmountable
   *  10: read-only
   *
   *                     First		Sector	   Last
   * Partition	Tag	Flags	Sector	Count		   Sector  Mount Directory
          2	     5 	01	    0		  20951040	 20951039
          3	    15	 01	    0		  15360		   15359
          4	    14	 01	  15360		20935680	 20951039
   

6. Create a VxVM disk group of all the converted disks and set alignment to 1.

   # /opt/VRTS/bin/vxdg -s init vxvmdg \
   hitachi_usp-vm0_031c hitachi_usp-vm0_0319 \
   hitachi_usp-vm0_0312 hitachi_usp-vm0_0311 cds=off
   
   # /opt/VRTS/bin/vxdg -g vxvmdg set alignment=1
   			

7. Create VxVM volumes in this disk group and set the permissions to the Oracle user.

   # /opt/VRTS/bin/vxassist -g vxvmdg make vxvmvol1 \
   hitachi_usp-vm0_031c 20935680
   # /opt/VRTS/bin/vxassist -g vxvmdg make vxvmvol2 \
   hitachi_usp-vm0_0319 20935680
   # /opt/VRTS/bin/vxassist -g vxvmdg make vxvmvol3 \
   hitachi_usp-vm0_0312 20935680
   # /opt/VRTS/bin/vxassist -g vxvmdg make vxvmvol4 \
   hitachi_usp-vm0_0311 20935680
   # /opt/VRTS/bin/vxedit -g vxvmdg set user=ora10gr2 \
   group=dba vxvmvol1
   # /opt/VRTS/bin/vxedit -g vxvmdg set user=ora10gr2 \
   group=dba vxvmvol2
   # /opt/VRTS/bin/vxedit -g vxvmdg set user=ora10gr2 \
   group=dba vxvmvol3
   # /opt/VRTS/bin/vxedit -g vxvmdg set user=ora10gr2 \
   group=dba vxvmvol4
   			

8. Use SQL to change the ASM disk string parameter. Log in to the ASM instance.

   $ export ORACLE_SID=+ASM
   $ sqlplus "/as sysasm"
   SQL> alter system set asm_diskstring='/dev/vx/rdsk/vxvm*/*' sid='*';
   System altered.
   

9. Verify that the disks are displayed in v$asm_disk.

   SQL> select name, state, header_status, path,
   TOTAL_MB, FREE_MB from v$asm_disk;
   NAME STATE    HEADER_STATU PATH                        TOTAL_MB FREE_MB
   ------------- ------------ --------------------------- -------- -------
        NORMAL   MEMBER       /dev/vx/rdsk/vxvmdg/vxvmvol1 10222   0
        NORMAL   MEMBER       /dev/vx/rdsk/vxvmdg/vxvmvol4 10222   0
        NORMAL   MEMBER       /dev/vx/rdsk/vxvmdg/vxvmvol3 10222   0
        NORMAL   MEMBER       /dev/vx/rdsk/vxvmdg/vxvmvol2 10222   0
   

10. Mount all the ASM disk groups on all the nodes of cluster.

    SQL> alter diskgroup data mount;
    Diskgroup altered.
    
    SQL> alter diskgroup fra mount;
    Diskgroup altered.
    
    SQL> select name, state,TOTAL_MB, FREE_MB, REQUIRED_MIRROR_FREE_MB,  \
    USABLE_FILE_MB  from v$asm_diskgroup;
    NAME   STATE   TOTAL_MB    FREE_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB
    -----  ------- ---------- ---------- ----------------------- --------------
    DATA   MOUNTED 20444      20349                       0          20349
    FRA    MOUNTED 20444      20349                       0          20349

11. Start the RAC database.

    $ srvctl start database -d PROD
    
    $ srvctl status database -d PROD
    Instance PROD1 is running on node punlabsm5001
    Instance PROD2 is running on node punlabsm5002
    
    SQL> select status from v$instance;
    
    STATUS
    ------------
    OPEN
    

To move OCR to VxVM volumes

1. As a root user, create a new VxVM disk group for OCR and vote storage.

   # vxdg -s init ocr_vote_dg <disk_device_name>

2. Create two VxVM volumes for OCR and vote storage.

   # vxassist -g ocr_vote_dg make ocrvol 1G
   # 	vxassist -g ocr_vote_dg make votevol 1G
   			

   Note:

   The OCR volume must be owned by the Oracle user, it must be in the dba group and its permissions must be set to 640. You must provide at least 100 MB of disk space for the OCR.

   Note:

   The voting volume must be owned by the Oracle user, it must be in the dba group and its permissions must be set to 644. You must provide at least 20 MB of disk space for the voting disk.

3. Ensure that there is a recent copy of the OCR file before making any changes.

   # ocrconfig -showbackup

4. If there is no recent backup copy of the OCR file, use the following command to generate an export of the online OCR file.

   # ocrconfig -export <OCR export_filename> -s online

5. If you need to recover by using this file, use the following command.

   # ocrconfig import <OCR export_filename>

   Note:

   You can now move or replace the OCR file to the new VxVM storage location.

6. To replace the OCR device with file name, provide the full path including file name.

   # ocrconfig -replace ocr <file name>

   For example:

   # ocrconfig -replace ocr /dev/vx/rdsk/ocr_vote_dg/ocrvol

To move voting disk to VxVM volumes

1. Determine the current voting disk location.

   # crsctl query css votedisk

2. Take a backup of all voting disks.

   # dd if=voting_disk_name of=backup_file_name

   To restore the voting disk from the backup file:

   # dd if=backup_file_name of=voting_disk_name

3. Move a voting disk by providing the full path including file name.

   # crsctl delete votedisk css <OLD_LOCATION> -force
   # crsctl add votedisk css <NEW_LOCATION> -force

   For example:

   # crsctl add votedisk css /dev/vx/rdsk/ocr_vote_dg/votevol

4. Start Oracle Clusterware on all the nodes, after modifying the voting disk.

   # crsctl start crs

5. Verify the voting disk location.

   # crsctl query css votedisk

To roll back to ASM on raw partitions

1. Shut down the Oracle RAC database and unmount all the ASM disk groups from each node of the cluster.

2. Destroy the VxVM disk group created during conversion.

   # /opt/VRTS/bin/vxdg destroy <disk_group_name>

3. Execute the vxdiskunsetup command on the disks.

   # /opt/VRTS/bin/vxdiskunsetup <disk_device_name>

   Note:

   If the vxdisksetup operation failed while doing conversion then using vxdiskunsetup might result in an error that the disk is not a VxVM disk. This error indicates that the vxdisksetup operation did not complete fully and it can be ignored.

4. Use the format command to remove all slices on the disks that vxdisksetup might have created.

5. Recreate the slices that existed before by using the prtvtoc output that you saved for all disks before you started the conversion process.

6. Change the ownership and permissions of the slices as before.

7. Mount the ASM disk groups on all nodes of the cluster.

8. Start the Oracle RAC database.