2. Configure the First Virtual Machine

To install VMware Tools from X with the RPM installer:

1. Choose [VM -> Install VMware Tools]. The guest operating system mounts the VMware Tools installation virtual CD.

2. Double-click the VMware Tools CD icon on the desktop..

3. Double-click the RPM installer in the root of the CD-ROM. VMwareTools-1.0.1-29996.i386.rpm

4. Enter the root password (if prompted).

5. Click [Continue]. The installer prepares the packages.

6. Click [Continue] when the installer presents a dialog box saying "Completed System Preparation". A dialog appears for Updating system, with a progress bar. When the installer is done, VMware Tools are installed. There is no confirmation or finish button.

7. In an X terminal, as root (su -), configure VMware Tools.

   # vmware-config-tools.pl

   Respond to any questions the installer displays on the screen. At the end of the configuration process, the program asks for the new screen resolution. You should pick the same screen resolution you selected during the CentOS Enterprise Linux install. I used option 2 ("800x600").

   NOTE: Be sure to respond [yes] if the installer offers to run the configuration program.

8. After the configuration is complete, you must now reboot the virtual machine.

   # init 6

9. After the virtual machine is rebooted and you log back in, you will notice that you no longer have to hit Ctrl-Alt to move between the virtual machine and the host operating system. Also, you will see that the mouse works more smoothly.

10. Once you have verified the VMware Tools is installed and configured correctly, unmount the VMware Tools installation virtual CD. Choose [VM -> Cancel VMware Tools Install].

Error while copying directory 
    /u01/app/oracle/product/crs with exclude file list 'null' to nodes 'rac2'.
[PRKC-1002 : All the submitted commands did not execute successfully]
---------------------------------------------
rac2:
   /bin/tar: ./bin/lsnodes: time stamp 2007-02-19 09:21:34 is 735 s in the future
   /bin/tar: ./bin/olsnodes: time stamp 2007-02-19 09:21:34 is 735 s in the future
   ...(more errors on this node)

Please note that although this would seem like a severe error from the OUI, it can safely be disregarded as a warning. The "tar" command DOES actually extract the files; however, when you perform a listing of the files (using ls -l) on the remote node, they will be missing the time field until the time on the server is greater than the timestamp of the file.
Before starting any of the above noted installations, ensure that each member node of the cluster is set as closely as possible to the same date and time. Oracle strongly recommends using the Network Time Protocol feature of most operating systems for this purpose, with all nodes using the same reference Network Time Protocol server.
Accessing a Network Time Protocol server, however, may not always be an option. In this case, when manually setting the date and time for the nodes in the cluster, ensure that the date and time of the node you are performing the software installations from (rac1) is less than all other nodes in the cluster (rac2). I generally use a 20 second difference as shown in the following example:

Setting the date and time from rac1:

# date -s "2/19/2007 23:00:00"

Setting the date and time from rac2:

# date -s "2/19/2007 23:00:20"

The two-node RAC configuration described in this article does not make use of a Network Time Protocol server.

• libaio-0.3.105-2.i386.rpm
• openmotif21-2.1.30-11.RHEL4.6.i386.rpm

• Installing Oracle10g Release 2 requires a minimum of 512MB of memory.
  (An inadequate amount of swap during the installation will cause the Oracle Universal Installer to either "hang" or "die")

• The check the amount of memory you have, type:

  # cat /proc/meminfo | grep MemTotal
  MemTotal:       755284 kB

• To check the amount of swap you have allocated, type:

  # cat /proc/meminfo | grep SwapTotal
  SwapTotal:     1540088 kB

• If you have less than 512MB of memory (between your RAM and SWAP), you can add temporary swap space by creating a temporary swap file. This way you do not have to use a raw device or even more drastic, rebuild your system.

  As root, make a file that will act as additional swap space, let's say about 300MB:
  # dd if=/dev/zero of=tempswap bs=1k count=300000

  Now we should change the file permissions:
  # chmod 600 tempswap

  Finally we format the "partition" as swap and add it to the swap space:
  # mke2fs tempswap
  # mkswap tempswap
  # swapon tempswap

Shared memory allows processes to access common structures and data by placing them in a shared memory segment. This is the fastest form of Inter-Process Communications (IPC) available - mainly due to the fact that no kernel involvement occurs when data is being passed between the processes. Data does not need to be copied between processes.
Oracle makes use of shared memory for its Shared Global Area (SGA) which is an area of memory that is shared by all Oracle backup and foreground processes. Adequate sizing of the SGA is critical to Oracle performance since it is responsible for holding the database buffer cache, shared SQL, access paths, and so much more.
To determine all shared memory limits, use the following:

# ipcs -lm

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 32768
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1

Setting SHMMAX

The SHMMAX parameters defines the maximum size (in bytes) for a shared memory segment. The Oracle SGA is comprised of shared memory and it is possible that incorrectly setting SHMMAX could limit the size of the SGA. When setting SHMMAX, keep in mind that the size of the SGA should fit within one shared memory segment. An inadequate SHMMAX setting could result in the following:

ORA-27123: unable to attach to shared memory segment

You can determine the value of SHMMAX by performing the following:

# cat /proc/sys/kernel/shmmax
33554432

The default value for SHMMAX is 32MB. This is often too small to configure the Oracle SGA. I generally set the SHMMAX parameter to 2GB using the following methods:

• You can alter the default setting for SHMMAX without rebooting the machine by making the changes directly to the /proc file system (/proc/sys/kernel/shmmax) by using the following command:

  # sysctl -w kernel.shmmax=2147483648

• You should then make this change permanent by inserting the kernel parameter in the /etc/sysctl.conf startup file:

  # echo "kernel.shmmax=2147483648" >> /etc/sysctl.conf

Setting SHMMNI

We now look at the SHMMNI parameters. This kernel parameter is used to set the maximum number of shared memory segments system wide. The default value for this parameter is 4096.
You can determine the value of SHMMNI by performing the following:

# cat /proc/sys/kernel/shmmni
4096

The default setting for SHMMNI should be adequate for our Oracle10g Release 2 RAC installation.

Setting SHMALL

Finally, we look at the SHMALL shared memory kernel parameter. This parameter controls the total amount of shared memory (in pages) that can be used at one time on the system. In short, the value of this parameter should always be at least:

ceil(SHMMAX/PAGE_SIZE)

The default size of SHMALL is 2097152 and can be queried using the following command:

# cat /proc/sys/kernel/shmall
2097152

The default setting for SHMALL should be adequate for our Oracle10g Release 2 RAC installation.

The page size in Red Hat Linux on the i386 platform is 4096 bytes. You can, however, use bigpages which supports the configuration of larger memory page sizes.

Setting Semaphores

Now that we have configured our shared memory settings, it is time to take care of configuring our semaphores. The best way to describe a semaphore is as a counter that is used to provide synchronization between processes (or threads within a process) for shared resources like shared memory. Semaphore sets are supported in System V where each one is a counting semaphore. When an application requests semaphores, it does so using "sets".

To determine all semaphore limits, use the following:

# ipcs -ls

------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767

You can also use the following command:

# cat /proc/sys/kernel/sem
250     32000   32      128

Setting SEMMSL

The SEMMSL kernel parameter is used to control the maximum number of semaphores per semaphore set.
Oracle recommends setting SEMMSL to the largest PROCESS instance parameter setting in the init.ora file for all databases on the Linux system plus 10. Also, Oracle recommends setting the SEMMSL to a value of no less than 100.

Setting SEMMNI

The SEMMNI kernel parameter is used to control the maximum number of semaphore sets in the entire Linux system.
Oracle recommends setting the SEMMNI to a value of no less than 100.

Setting SEMMNS

The SEMMNS kernel parameter is used to control the maximum number of semaphores (not semaphore sets) in the entire Linux system.
Oracle recommends setting the SEMMNS to the sum of the PROCESSES instance parameter setting for each database on the system, adding the largest PROCESSES twice, and then finally adding 10 for each Oracle database on the system.
Use the following calculation to determine the maximum number of semaphores that can be allocated on a Linux system. It will be the lesser of:

SEMMNS  -or-  (SEMMSL * SEMMNI)

Setting SEMOPM

The SEMOPM kernel parameter is used to control the number of semaphore operations that can be performed per semop system call.
The semop system call (function) provides the ability to do operations for multiple semaphores with one semop system call. A semaphore set can have the maximum number of SEMMSL semaphores per semaphore set and is therefore recommended to set SEMOPM equal to SEMMSL.
Oracle recommends setting the SEMOPM to a value of no less than 100.

Setting Semaphore Kernel Parameters

Finally, we see how to set all semaphore parameters. In the following, the only parameter I care about changing (raising) is SEMOPM. All other default settings should be sufficient for our example installation.

• You can alter the default setting for all semaphore settings without rebooting the machine by making the changes directly to the /proc file system (/proc/sys/kernel/sem) by using the following command:

  # sysctl -w kernel.sem="250 32000 100 128"

• You should then make this change permanent by inserting the kernel parameter in the /etc/sysctl.conf startup file:

  # echo "kernel.sem=250 32000 100 128" >> /etc/sysctl.conf

Setting File Handles

When configuring the Red Hat Linux server, it is critical to ensure that the maximum number of file handles is large enough. The setting for file handles denotes the number of open files that you can have on the Linux system.
Use the following command to determine the maximum number of file handles for the entire system:

# cat /proc/sys/fs/file-max
102563

Oracle recommends that the file handles for the entire system be set to at least 65536.

• You can alter the default setting for the maximum number of file handles without rebooting the machine by making the changes directly to the /proc file system (/proc/sys/fs/file-max) using the following:

  # sysctl -w fs.file-max=65536

• You should then make this change permanent by inserting the kernel parameter in the /etc/sysctl.conf startup file:

  # echo "fs.file-max=65536" >> /etc/sysctl.conf

You can query the current usage of file handles by using the following: # cat /proc/sys/fs/file-nr 825 0 65536 The file-nr file displays three parameters: Total allocated file handles Currently used file handles Maximum file handles that can be allocated

If you need to increase the value in /proc/sys/fs/file-max, then make sure that the ulimit is set properly. Usually for Linux 2.4 and 2.6 it is set to unlimited. Verify the ulimit setting my issuing the ulimit command: # ulimit unlimited

Setting IP Local Port Range

Configure the system to allow a local port range of 1024 through 65000.
Use the following command to determine the value of ip_local_port_range:

# cat /proc/sys/net/ipv4/ip_local_port_range
32768   61000

The default value for ip_local_port_range is ports 32768 through 61000. Oracle recommends a local port range of 1024 to 65000.

• You can alter the default setting for the local port range without rebooting the machine by making the changes directly to the /proc file system (/proc/sys/net/ipv4/ip_local_port_range) by using the following command:

  # sysctl -w net.ipv4.ip_local_port_range="1024 65000"

• You should then make this change permanent by inserting the kernel parameter in the /etc/sysctl.conf startup file:

  # echo "net.ipv4.ip_local_port_range = 1024 65000" >> /etc/sysctl.conf

$ /sbin/ifconfig -a
eth0      Link encap:Ethernet  HWaddr 00:0C:29:07:E6:0B
          inet addr:192.168.1.111  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe07:e60b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:170 errors:0 dropped:0 overruns:0 frame:0
          TX packets:146 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:14360 (14.0 KiB)  TX bytes:11875 (11.5 KiB)
          Interrupt:185 Base address:0x1400

eth1      Link encap:Ethernet  HWaddr 00:0C:29:07:E6:15
          inet addr:192.168.2.111  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe07:e615/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:197 errors:0 dropped:0 overruns:0 frame:0
          TX packets:21 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:14618 (14.2 KiB)  TX bytes:1386 (1.3 KiB)
          Interrupt:169 Base address:0x1480

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:1962 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1962 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:3226318 (3.0 MiB)  TX bytes:3226318 (3.0 MiB)

sit0      Link encap:IPv6-in-IPv4
          NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

About Virtual IP

It's all about availability of the application. When a node fails, the VIP associated with it is supposed to be automatically failed over to some other node. When this occurs, two things happen.

1. The new node re-arps the world indicating a new MAC address for the address. For directly connected clients, this usually causes them to see errors on their connections to the old address.

2. Subsequent packets sent to the VIP go to the new node, which will send error RST packets back to the clients. This results in the clients getting errors immediately.

This means that when the client issues SQL to the node that is now down, or traverses the address list while connecting, rather than waiting on a very long TCP/IP time-out (~10 minutes), the client receives a TCP reset. In the case of SQL, this is ORA-3113. In the case of connect, the next address in tnsnames is used.

Going one step further is making use of Transparent Application Failover (TAF). With TAF successfully configured, it is possible to completely avoid ORA-3113 errors alltogether!

Without using VIPs, clients connected to a node that died will often wait a 10 minute TCP timeout period before getting an error. As a result, you don't really have a good HA solution without using VIPs.

Source - Metalink: "RAC Frequently Asked Questions" (Note:220970.1)

Disk 1

fdisk /dev/sdb
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-261, default 1): 1
Last cylinder or +size or +sizeM or +sizeK (1-261, default 261): 261

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Disk 2

fdisk /dev/sdc
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1566, default 1): 1
Last cylinder or +size or +sizeM or +sizeK (1-1566, default 1566): 1566

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Disk 3

fdisk /dev/sdd
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1566, default 1): 1
Last cylinder or +size or +sizeM or +sizeK (1-1566, default 1566): 1566

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Disk 4

fdisk /dev/sde
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1566, default 1): 1
Last cylinder or +size or +sizeM or +sizeK (1-1566, default 1566): 1566

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

fdisk -l

Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1         910     7309543+  83  Linux
/dev/sda2             911        1170     2088450   82  Linux swap
/dev/sda3            1171        2610    11566800   83  Linux

Disk /dev/sdb: 536 MB, 536870912 bytes
64 heads, 32 sectors/track, 512 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1         512      524272   83  Linux

Disk /dev/sdc: 3221 MB, 3221225472 bytes
255 heads, 63 sectors/track, 391 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1         391     3140676   83  Linux

Disk /dev/sdd: 3221 MB, 3221225472 bytes
255 heads, 63 sectors/track, 391 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1         391     3140676   83  Linux

Disk /dev/sde: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1         261     2096451   83  Linux

To avoid this problem, you must modify these files to suppress all output on STDERR as in the following examples:

• Bourne, Bash, or Korn shell:

  if [ -t 0 ]; then
      stty intr ^C
  fi

• C shell:

  test -t 0
  if ($status == 0) then
      stty intr ^C
  endif

The rsh daemon validates users using the /etc/hosts.equiv file or the .rhosts file found in the user's (oracle's) home directory.

First, let's make sure that we have the rsh RPMs installed on both of the Oracle RAC nodes in the cluster:

# rpm -q rsh rsh-server
rsh-0.17-25.4
rsh-server-0.17-25.4

If rsh is not installed, run the following command from the CD where the RPM is located: # su - # rpm -ivh rsh-0.17-25.4.i386.rpm rsh-server-0.17-25.4.i386.rpm

To enable the "rsh" and "rlogin" services, the "disable" attribute in the /etc/xinetd.d/rsh file must be set to "no" and xinetd must be reloaded. This can be done by running the following commands on both Oracle RAC nodes in the cluster:

# su -
# chkconfig rsh on
# chkconfig rlogin on
# service xinetd reload
Reloading configuration: [  OK  ]

# su -
# touch /etc/hosts.equiv
# chmod 600 /etc/hosts.equiv
# chown root.root /etc/hosts.equiv

# cat /etc/hosts.equiv
+rac1 oracle
+rac2 oracle
+rac1-priv oracle
+rac2-priv oracle

In the above example, the second field permits only the oracle user account to run rsh commands on the specified nodes. For security reasons, the /etc/hosts.equiv file should be owned by root and the permissions should be set to 600. In fact, some systems will only honor the content of this file if the owner of this file is root and the permissions are set to 600.

Before attempting to test your rsh command, ensure that you are using the correct version of rsh. By default, Red Hat Linux puts /usr/kerberos/sbin at the head of the $PATH variable. This will cause the Kerberos version of rsh to be executed. I will typically rename the Kerberos version of rsh so that the normal rsh command will be used. Use the following: # su - # which rsh /usr/kerberos/bin/rsh # mv /usr/kerberos/bin/rsh /usr/kerberos/bin/rsh.original # mv /usr/kerberos/bin/rcp /usr/kerberos/bin/rcp.original # mv /usr/kerberos/bin/rlogin /usr/kerberos/bin/rlogin.original # which rsh /usr/bin/rsh

You should now test your connections and run the rsh command from the node that will be performing the Oracle Clusterware and 10g RAC installation. I will be using the node rac1 to perform all installs so this is where I will run the following commands from:

# su - oracle

$ rsh rac1 ls -l /etc/hosts.equiv
-rw-------  1 root root 78 Feb 19 18:28 /etc/hosts.equiv

$ rsh rac1-priv ls -l /etc/hosts.equiv
-rw-------  1 root root 78 Feb 19 18:28 /etc/hosts.equiv

$ rsh rac2 ls -l /etc/hosts.equiv
-rw-------  1 root root 78 Feb 19 18:28 /etc/hosts.equiv

$ rsh rac2-priv ls -l /etc/hosts.equiv
-rw-------  1 root root 78 Feb 19 18:28 /etc/hosts.equiv

Unlike when using secure shell, no other actions or commands are needed to enable user equivalence using the remote shell. User equivalence will be enabled for the "oracle" UNIX user account after successfully logging in to a terminal session.

5. Configure Oracle Automatic Storage Management (ASM)

# init 6

• NM: Node Manager that keep track of all the nodes in the cluster.conf
• HB: Heartbeat service that issues up/down notifications when nodes join or leave the cluster
• TCP: Handles communication between the nodes
• DLM: Distributed lock manager that keeps track of all locks, its owners, and status
• CONFIGFS: User space driven configuration file system mounted at /config
• DLMFS: User space interface to the kernel space DLM

# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/sda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
configfs on /config type configfs (rw)
ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)
/dev/sdb1 on /ocfs type ocfs2 (rw,_netdev,datavolume,nointr,heartbeat=local)

Now, let's make sure that the ocfs2.ko kernel module is being loaded and that the file system will be mounted during the boot process.
If you have been following along with the examples in this article, the actions to load the kernel module and mount the OCFS2 file system should already be enabled. However, we should still check those options by running the following on both nodes in the RAC cluster as the root user account:

rac1-> su -
# chkconfig --list o2cb
o2cb            0:off   1:off   2:on    3:on    4:on    5:on    6:off

# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/sda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
configfs on /config type configfs (rw)
ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)
/dev/sdb1 on /ocfs type ocfs2 (rw,_netdev,datavolume,nointr,heartbeat=local)

If you modified the O2CB heartbeat threshold, you should verify that it is set correctly:

# cat /proc/fs/ocfs2_nodemanager/hb_dead_threshold
601

# cat /proc/fs/ocfs2/version
OCFS2 1.2.4 Thu Feb  1 15:04:31 PST 2007 (build a821ad1645e42e94b93ec4904c40dd10)

  Oracle Clusterware Release 2 (10.2.0.1.0)

• 10201_clusterware_linux32.zip   (228,239,016 bytes)

Next, we need to download the Oracle Database 10g Release 2 (10.2.0.1.0) Software for Linux x86. This can be downloaded from the same page used to download the Oracle Clusterware Release 2 software:

  Oracle Database 10g Release 2 (10.2.0.1.0)

• 10201_database_linux32.zip   (668,734,007 bytes)

Extract the Clusterware package as follows:

# su - oracle
rac1-> cd /u01/staging
rac1-> unzip 10201_clusterware_linux32.zip

Then extract the Oracle10g Database Software:

rac1-> cd /u01/staging
rac1-> unzip 10201_database_linux32.zip

Check Required RPMs

binutils-2.15.92.0.2-21
compat-db-4.1.25-9
compat-gcc-32-3.2.3-47.3
compat-gcc-32-c++-3.2.3-47.3
compat-libstdc++-33-3.2.3-47.3
compat-libgcc-296-2.96-132.7.2
control-center-2.8.0-12.rhel4.5
cpp-3.4.6-3
gcc-3.4.6-3
gcc-c++-3.4.6-3
glibc-2.3.4-2.25
glibc-common-2.3.4-2.25
glibc-devel-2.3.4-2.25
glibc-headers-2.3.4-2.25
glibc-kernheaders-2.4-9.1.98.EL
gnome-libs-1.4.1.2.90-44.1
libaio-0.3.105-2
libstdc++-3.4.6-3
libstdc++-devel-3.4.6-3
make-3.80-6.EL4
openmotif-2.2.3-10.RHEL4.5
openmotif21-2.1.30-11.RHEL4.6 
pdksh-5.2.14-30.3
setarch-1.6-1
sysstat-5.0.5-11.rhel4
xscreensaver-4.18-5.rhel4.11

Note that the openmotif RPM packages are only required to install Oracle demos. This article does not cover the installation of Oracle demos.

To query package information (gcc and glibc-devel for example), use the "rpm -q <PackageName> [, <PackageName>]" command as follows:

# rpm -q gcc glibc-devel
gcc-3.4.6-3
glibc-devel-2.3.4-2.25

# rpm -Uvh gcc-3.4.6-3.i386.rpm

JDK 1.4.2

You must have JDK 1.4.2 installed on your system before you can run CVU. If you do not have JDK 1.4.2 installed on your system, and you attempt to run CVU, you will receive an error message similar to the following:

ERROR. Either CV_JDKHOME environment variable should be set
or /stagepath/cluvfy/jrepack.zip should exist.

If you do not have JDK 1.4.2 installed, then download it from the Sun Web site, and use the Sun instructions to install it. JDK 1.4.2 is available as a download from the following Web site: http://www.sun.com/java.

If you do have JDK 1.4.2 installed, then you must define the user environment variable CV_JDKHOME for the path to the JDK. For example, if JDK 1.4.2 is installed in /usr/local/j2re1.4.2_08, then log in as the user that you plan to use to run CVU, and enter the following commands:

CV_JDKHOME=/usr/local/j2re1.4.2_08
export CV_JDKHOME

Install cvuqdisk RPM (RHEL Users Only) in both nodes

The second pre-requisite for running the CVU is for Red Hat Linux users. If you are using Red Hat Linux, then you must download and install the Red Hat operating system package cvuqdisk to both of the Oracle RAC nodes in the cluster. This means you will need to install the cvuqdisk RPM to both rac1 and rac2. Without cvuqdisk, CVU will be unable to discover shared disks, and you will receive the error message "Package cvuqdisk not installed" when you run CVU.

The cvuqdisk RPM can be found on the Oracle Clusterware installation media in the rpm directory. For the purpose of this article, the Oracle Clusterware media was extracted to the /u01/staging/clusterware directory on  rac1. Note that before installing the cvuqdisk RPM, we need to set an environment variable named CVUQDISK_GRP to point to the group that will own the cvuqdisk utility. The default group is oinstall which is not the group we are using for the oracle UNIX user account in this article. Since we are using the dba group, we will need to set CVUQDISK_GRP=dba before attempting to install the cvuqdisk RPM.

Locate and copy the cvuqdisk RPM from rac1 to rac2 then perform the following steps on both Oracle RAC nodes to install:

$ su -
# cd /u01/staging/clusterware/rpm
# CVUQDISK_GRP=dba; export CVUQDISK_GRP

# rpm -iv cvuqdisk-1.0.1-1.rpm
Preparing packages for installation...
cvuqdisk-1.0.1-1

# ls -l /usr/sbin/cvuqdisk
-rwsr-x---  1 root dba 4168 Jun  2  2005 /usr/sbin/cvuqdisk

Verify Remote Access / User Equivalence

The CVU should be run from rac1— the node we will be performing all of the Oracle installations from. Before running CVU, login as the oracle user account and verify remote access / user equivalence is configured to all nodes in the cluster. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for each key that you generated when prompted:

# su - oracle
rac1-> exec /usr/bin/ssh-agent $SHELL
rac1-> /usr/bin/ssh-add
Enter passphrase for /u01/app/oracle/.ssh/id_rsa: xxxxx
Identity added: /u01/app/oracle/.ssh/id_rsa (/u01/app/oracle/.ssh/id_rsa)
Identity added: /u01/app/oracle/.ssh/id_dsa (/u01/app/oracle/.ssh/id_dsa)

When using the remote shell method, user equivalence is generally defined in the /etc/hosts.equiv file for the oracle user account and is enabled on all new terminal shell sessions.

Once all prerequisites for using CVU have been met, we can start by checking that all pre-installation tasks for Oracle Clusterware (CRS) are completed by executing the following command as the "oracle" UNIX user account (with user equivalence enabled) from rac1:

rac1-> cd /u01/staging/clusterware/cluvfy
rac1-> ./runcluvfy.sh stage -pre crsinst -n rac1,rac2 -verbose | more

Review the CVU report. Note that there are several errors you may ignore in this report.

The first error is with regards to membership of the user "oracle" in group "oinstall" [as Primary]. For the purpose of this article, the "oracle" user account will only be assigned to the "dba" group so this error can be safely ignored.

The second error is with regards to finding a suitable set of interfaces for VIPs. This is a bug documented in Metalink Note 338924.1:

Suitable interfaces for the private interconnect on subnet "192.168.1.0":
rac2 eth0:192.168.2.132
rac1 eth0:192.168.2.131

Suitable interfaces for the private interconnect on subnet "192.168.2.0":
rac2 eth1:192.168.2.32
rac1 eth1:192.168.2.31

ERROR:
Could not find a suitable set of interfaces for VIPs.

Result: Node connectivity check failed.

As documented in the note, this error can be safely ignored.

The last set of errors that can be ignored deal with specific RPM package versions that do not exist in this Linux version For example:

• compat-gcc-7.3-2.96.128
• compat-gcc-c++-7.3-2.96.128
• compat-libstdc++-7.3-2.96.128
• compat-libstdc++-devel-7.3-2.96.128

While these specific packages are listed as missing in the CVU report, please ensure that the correct versions of the compat-* packages are installed on both of the Oracle RAC nodes in the cluster. For example, these would be:

• compat-gcc-32-3.2.3-47.3
• compat-gcc-32-c++-3.2.3-47.3
• compat-libstdc++-33-3.2.3-47.3

The next CVU check to run will verify the hardware and operating system setup. Again, run the following as the "oracle" UNIX user account from rac1:

$ cd /u01/staging/clusterware/cluvfy
$ ./runcluvfy.sh stage -post hwos -n rac1,rac2 -verbose | more

Review the CVU report. As with the previous check (pre-installation tasks for CRS), the check for finding a suitable set of interfaces for VIPs will fail and can be safely ignored.

Also note that the check for shared storage accessibility will fail.

Checking shared storage accessibility...

WARNING:
Unable to determine the sharedness of /dev/sdf on nodes:
        rac2,rac2,rac2,rac2,rac2,rac1,rac1,rac1,rac1,rac1


Shared storage check failed on nodes "rac2,rac1".

This too can be safely ignored. While we know the disks are visible and shared from both of our Oracle RAC nodes in the cluster, the check itself fails. Several reasons for this have been documented. The first came from Metalink indicating that cluvfy currently does not work with devices other than SCSI devices. This would include devices like EMC PowerPath and volume groups like those from Openfiler. At the time of this writing, no workaround exists other than to use manual methods for detecting shared devices. Another reason for this error was documented by Bane Radulovic at Oracle Corporation. His research shows that CVU calls smartclt on Linux, and the problem is that smartclt does not return the serial number from our virtual SCSI devices. The virtual SCSI devices from VMware so not support SMART. For example, a check against /dev/sdc shows:

# /usr/sbin/smartctl -i /dev/sdc
smartctl version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

Device: VMware,  VMware Virtual S Version: 1.0
Device type: disk
Local Time is: Mon Feb 19 19:48:15 2007 EST
Device does not support SMART

So, what exactly is the Oracle Clusterware responsible for? It contains all of the cluster and database configuration metadata along with several system management features for RAC. It allows the DBA to register and invite an Oracle instance (or instances) to the cluster. During normal operation, Oracle Clusterware will send messages (via a special ping operation) to all nodes configured in the cluster - often called the heartbeat. If the heartbeat fails for any of the nodes, it checks with the Oracle Clusterware configuration files (on the shared disk) to distinguish between a real node failure and a network failure.

After installing Oracle Clusterware, the Oracle Universal Installer (OUI) used to install the Oracle10g database software (next section) will automatically recognize these nodes. Like the Oracle Clusterware install we will be performing in this section, the Oracle10g database software only needs to be run from one node. The OUI will copy the software packages to all nodes configured in the RAC cluster.

Oracle Clusterware Shared Files

The two shared files used by Oracle Clusterware will be stored on the Oracle Cluster File System, Release 2 (OFCS2) we created earlier. The two shared Oracle Clusterware files are:

• Oracle Cluster Registry (OCR)
  • File 1 : /ocfs/OCRFile
  • File 2 : /ocfs/OCRFile_mirror
  • Size : (2 * 100MB) = 200M

• CRS Voting Disk
  • File 1 : /ocfs/CSSFile
  • File 2 : /ocfs/CSSFile_mirror1
  • File 3 : /ocfs/CSSFile_mirror2
  • Size : (3 * 20MB) = 60MB

It is not possible to use Automatic Storage Management (ASM) for the two shared Oracle Clusterware files: Oracle Cluster Registry (OCR) or the CRS Voting Disk files. The problem is that these files need to be in place and accessible BEFORE any Oracle instances can be started. For ASM to be available, the ASM instance would need to be run first.

The two shared files could be stored on the OCFS2, shared RAW devices, or another vendor's clustered file system.

Verifying Terminal Shell Environment

# hostname
rac1

# xhost +
access control disabled, clients can connect from any host

Verify Remote Access / User Equivalence
# su - oracle
    Verify you are able to run the Secure Shell commands (ssh or scp) or the Remote Shell commands (rsh and rcp) on the Linux server you will be running the Oracle Universal Installer from against all other Linux servers in the cluster without being prompted for a password.
    When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the OUI. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for each key that you generated when prompted:
rac1-> exec /usr/bin/ssh-agent $SHELL
rac1-> /usr/bin/ssh-add

1. Welcome: Click on Next.
2. Specify Inventory directory and credentials:

Enter the full path of the inventory directory: /u01/app/oracle/oraInventory.
Specify Operating System group name: oinstall.

3. Specify Home Details:

Name: OraCrs10g_home
/u01/app/oracle/product/10.2.0/crs_1

4. Product-Specific Prerequisite Checks:

Ignore the warning on physical memory requirement.

5. Specify Cluster Configuration: Click on Add.

Public Node Name: rac2.mycorpdomain.com
Private Node Name: rac2-priv.mycorpdomain.com
Virtual Host Name: rac2-vip.mycorpdomain.com

6. Specify Network Interface Usage:

Interface Name: eth0
Subnet: 192.168.2.0
Interface Type: Public
Interface Name: eth1
Subnet: 10.10.10.0
Interface Type: Private

7. Specify Oracle Cluster Registry (OCR) Location: Select External Redundancy.

Specify OCR Location: /ocfs/clusterware/ocr

8. Specify Voting Disk Location: Select External Redundancy.
Similarly, for simplicity, we have chosen not to mirror the Voting Disk.

Voting Disk Location: /ocfs/clusterware/votingdisk

9. Summary: Click on Install.
10.   Execute Configuration scripts: Execute the scripts below as the root user sequentially, one at a time. Do not proceed to the next script until the current script completes.

Execute /u01/app/oracle/oraInventory/orainstRoot.sh on rac1 as root.
Execute /u01/app/oracle/oraInventory/orainstRoot.sh on rac2 as root.
Execute /u01/app/oracle/product/10.2.0/crs_1/root.sh on rac1 as root.
Execute /u01/app/oracle/product/10.2.0/crs_1/root.sh on rac2 as root.

The root.sh script on rac2 invoked the VIPCA automatically but it failed with the error "The given interface(s), "eth0" is not public. Public interfaces should be used to configure virtual IPs."
As you are using a non-routable IP address (192.168.x.x) for the public interface, the Oracle Cluster Verification Utility (CVU) could not find a suitable public interface. A workaround (noted in Metalink article 338924.1) is to
run VIPCA manually.
11.   As the root user, manually invokes VIPCA on the second node.

# /u01/app/oracle/product/10.2.0/crs_1/bin/vipca

12.   Welcome: Click on Next.
13.   Network Interfaces: Select eth0.

14.   Virtual IPs for cluster nodes:

Node name: rac1
IP Alias Name: rac1-vip
IP address: 192.168.2.31
Subnet Mask: 255.255.255.0
Node name: rac2
IP Alias Name: rac2-vip
IP address: 192.168.2.32
Subnet Mask: 255.255.255.0

15.   Summary: Click on Finish

After the installation of Oracle Clusterware, we can run through several tests to verify the install was successful. Run the following commands on all nodes in the RAC cluster.

rac1-> /u01/app/oracle/product/10.2.0/crs_1/bin/cluvfy stage -post crsinst -n rac1,rac2 | more

Check cluster nodes

rac1-> /u01/app/oracle/product/crs_1/bin/olsnodes -n
rac1        1
rac2        2

Check Oracle Clusterware Auto-Start Scripts

rac1-> ls -l /etc/init.d/init.*
-r-xr-xr-x  1 root root  1951 Feb 19 20:05 /etc/init.d/init.crs
-r-xr-xr-x  1 root root  4714 Feb 19 20:05 /etc/init.d/init.crsd
-r-xr-xr-x  1 root root 35394 Feb 19 20:05 /etc/init.d/init.cssd
-r-xr-xr-x  1 root root  3190 Feb 19 20:05 /etc/init.d/init.evmd

8. Install Oracle Database 10g Release 2

rac1-> cd /u01/staging/clusterware/cluvfy
rac1-> ./runcluvfy.sh stage -pre dbinst -n rac1,rac2 -r 10gR2 -verbose | more

rac1-> /u01/staging/database/runInstaller

1. Welcome: Click on Next.
2. Select Installation Type:

Select Enterprise Edition.

3. Specify Home Details:

Name: OraDb10g_home1
Path: /u01/app/oracle/product/10.2.0/db_1

4. Specify Hardware Cluster Installation Mode:

Select Cluster Installation.
Click on Select All. to select all servers: rac1 and rac2

5. Product-Specific Prerequisite Checks:

Ignore the warning on physical memory requirement.

6. Select Configuration Option:

Select: Install database Software only.
We will create the clustered database as a separate step using dbca

7. Summary: Click on Install.
8. Root Script Window: Execute the scripts below as the root user in all nodes.

Execute /u01/app/oracle/product/10.2.0/db_1/root.sh on rac1 as root.

Execute /u01/app/oracle/product/10.2.0/db_1/root.sh on rac2 as root.

9.   Return to the Execute Configuration scripts screen on rac1 and click on OK.
10.   End of Installation: Click on Exit.

Perform the following configuration procedures from only one of the Oracle RAC nodes in the cluster (rac1)! The Network Configuration Assistant (NETCA) will setup the TNS listener in a clustered configuration on both of Oracle RAC nodes in the cluster.

The Database Configuration Assistant (DBCA) requires the Oracle TNS Listener process to be configured and running on all nodes in the RAC cluster before it can create the clustered database.
The process of creating the TNS listener only needs to be performed from one of the nodes in the RAC cluster. All changes will be made and replicated to both Oracle RAC nodes in the cluster. On one of the nodes (I will be using rac1) bring up the Network Configuration Assistant (NETCA) and run through the process of creating a new TNS listener process and to also configure the node for local access.

To start the NETCA, run the following:

rac1-> netca &

The following screenshots walk you through the process of creating a new Oracle listener for our RAC environment.

Screen Name	Response
Select the Type of Oracle Net Services Configuration	Select Cluster configuration
Select the nodes to configure	Select all of the nodes: rac1 and rac2.
Type of Configuration	Select Listener configuration.
Listener Configuration - Next 6 Screens	The following screens are now like any other normal listener configuration. You can simply accept the default parameters for the next six screens:    What do you want to do: Add    Listener name: LISTENER    Selected protocols: TCP    Port number: 1521    Configure another listener: No    Listener configuration complete! [ Next ] You will be returned to this Welcome (Type of Configuration) Screen.
Type of Configuration	Select Naming Methods configuration.
Naming Methods Configuration	The following screens are:    Selected Naming Methods: Local Naming    Naming Methods configuration complete! [ Next ] You will be returned to this Welcome (Type of Configuration) Screen.
Type of Configuration	Click Finish to exit the NETCA.

The Oracle TNS listener process should now be running on both nodes in the RAC cluster:

rac1-> hostname
rac1

rac1-> ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_RAC1

=====================

rac1-> hostname
rac2

rac1-> ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_RAC2

The database creation process should only be performed from one of the Oracle RAC nodes in the cluster (rac1)!
We will be using the Oracle Database Configuration Assistant (DBCA) to create the clustered database.
Before executing the Database Configuration Assistant, make sure that $ORACLE_HOME and $PATH are set appropriately for the $ORACLE_BASE/product/10.2.0/db_1 environment.
You should also verify that all services we have installed up to this point (Oracle TNS listener, Oracle Clusterware processes, etc.) are running before attempting to start the clustered database creation process.

rac1-> cd /u01/staging/clusterware/cluvfy
rac1-> ./runcluvfy.sh stage -pre dbcfg -n rac1,rac2 -d ${ORACLE_HOME} -verbose | more

rac1->  dbca &

When the Oracle Database Configuration Assistant has completed, you will have a fully functional Oracle RAC cluster running!

• Adding and deleting node level applications.
• Setting and unsetting the environment for node-level applications.
• Administering node applications.
• Administering ASM instances.
• Starting and stopping a group of programs that includes virtual IP addresses, listeners, Oracle Notification Services, and Oracle Enterprise Manager agents (for maintenance purposes).

$ export ORACLE_SID=devdb1
$ emctl stop dbconsole
$ srvctl stop instance -d devdb -i devdb1
$ srvctl stop asm -n rac1
$ srvctl stop nodeapps -n rac1

Starting the Oracle10g RAC Environment

$ export ORACLE_SID=devdb1
$ srvctl start nodeapps -n rac1
$ srvctl start asm -n rac1
$ srvctl start instance -d devdb -i devdb1
$ emctl start dbconsole

SELECT inst_id, instance_number inst_no, instance_name inst_name, parallel,
  status, database_status db_status, active_state state, host_name host
FROM gv$instance
ORDER BY inst_id;
 INST_ID  INST_NO INST_NAME  PAR STATUS  DB_STATUS    STATE     HOST
-------- -------- ---------- --- ------- ------------ --------- --------
       1        1 devdb1     YES OPEN    ACTIVE       NORMAL    rac1
       2        2 devdb2     YES OPEN    ACTIVE       NORMAL    rac2

select name from v$datafile
union
select member from v$logfile
union
select name from v$controlfile
union
select name from v$tempfile;

NAME
-------------------------------------------
++RECOVERYDEST/devdb/controlfile/current.256.578676737
++RECOVERYDEST/devdb/onlinelog/group_1.257.578676745
++RECOVERYDEST/devdb/onlinelog/group_2.258.578676759
++RECOVERYDEST/devdb/onlinelog/group_3.259.578682963
++RECOVERYDEST/devdb/onlinelog/group_4.260.578682987
++DG1/devdb/controlfile/current.256.578676735
++DG1/devdb/datafile/example.263.578676853
++DG1/devdb/datafile/indx.270.578685723
++DG1/devdb/datafile/sysaux.261.578676829
++DG1/devdb/datafile/system.259.578676767
++DG1/devdb/datafile/undotbs1.260.578676809
++DG1/devdb/datafile/undotbs1.271.578685941
++DG1/devdb/datafile/undotbs2.264.578676867
++DG1/devdb/datafile/undotbs2.272.578685977
++DG1/devdb/datafile/users.265.578676887
++DG1/devdb/datafile/users.269.578685653
++DG1/devdb/onlinelog/group_1.257.578676739
++DG1/devdb/onlinelog/group_2.258.578676753
++DG1/devdb/onlinelog/group_3.266.578682951
++DG1/devdb/onlinelog/group_4.267.578682977
++DG1/devdb/tempfile/temp.262.578676841

select tablespace_name, file_name
from dba_data_files
union
select tablespace_name, file_name
from dba_temp_files;

TABLESPACE_NAME  FILE_NAME
---------------- --------------------------------------------------
EXAMPLE          +DG1/devdb/datafile/example.263.578676853
SYSAUX           +DG1/devdb/datafile/sysaux.261.578676829
SYSTEM           +DG1/devdb/datafile/system.259.578676767
TEMP             +DG1/devdb/tempfile/temp.262.578676841
UNDOTBS1         +DG1/devdb/datafile/undotbs1.260.578676809
UNDOTBS2         +DG1/devdb/datafile/undotbs2.264.578676867
USERS            +DG1/devdb/datafile/users.265.578676887

$ sqlplus "/ as sysdba"

create user scott identified by tiger default tablespace users;
grant dba, resource, connect to scott;

alter database datafile '+DG1/devdb/datafile/users.265.578676887' resize 500m;
alter tablespace users add datafile '+DG1' size 250m autoextend off;

create tablespace indx datafile '+DG1' size 250m
autoextend on next 50m maxsize unlimited
extent management local autoallocate
segment space management auto;

alter database datafile '+DG1/devdb/datafile/system.259.578676767' resize 800m;

alter database datafile '+DG1/devdb/datafile/sysaux.261.578676829' resize 500m;

alter tablespace undotbs1 add datafile '+DG1' size 250m
autoextend on next 50m maxsize 2048m;

alter tablespace undotbs2 add datafile '+DG1' size 250m
autoextend on next 50m maxsize 2048m;

alter database tempfile '+DG1/devdb/tempfile/temp.262.578676841' resize 1024m;

Here is a snapshot of the tablespaces I have defined for my test database environment:

Status    Tablespace Name TS Type      Ext. Mgt.  Seg. Mgt.    Tablespace Size    Used (in bytes) Pct. Used
--------- --------------- ------------ ---------- --------- ------------------ ------------------ ---------
ONLINE    UNDOTBS1       UNDO          LOCAL      MANUAL           471,859,200         62,586,880        13
ONLINE    SYSAUX         PERMANENT    LOCAL       AUTO             524,288,000        278,724,608        53
ONLINE    USERS          PERMANENT    LOCAL       AUTO             786,432,000            131,072         0
ONLINE    SYSTEM         PERMANENT    LOCAL       MANUAL           838,860,800        502,726,656        60
ONLINE    EXAMPLE        PERMANENT    LOCAL       AUTO             157,286,400         83,820,544        53
ONLINE    INDX           PERMANENT    LOCAL       AUTO             262,144,000             65,536         0
ONLINE    UNDOTBS2       UNDO          LOCAL      MANUAL           471,859,200          1,835,008         0
ONLINE    TEMP           TEMPORARY    LOCAL       MANUAL         1,073,741,824         27,262,976         3
                                                    ------------------ ------------------ ---------
avg                                                                                              23
sum                                                      4,586,471,424        957,153,280

8 rows selected.

LISTENERS_DEVDB =
  (ADDRESS_LIST =
    (ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip)(PORT = 1521))
    (ADDRESS = (PROTOCOL = TCP)(HOST = rac2-vip)(PORT = 1521))
  )

DEVDB2 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = rac2-vip)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = devdb)
      (INSTANCE_NAME = devdb2)
    )
  )

DEVDB1 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = devdb)
      (INSTANCE_NAME = devdb1)
    )
  )

CRM =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip)(PORT = 1521))
    (ADDRESS = (PROTOCOL = TCP)(HOST = rac2-vip)(PORT = 1521))
    (LOAD_BALANCE = yes)
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = CRM)
      (FAILOVER_MODE =
        (TYPE = SELECT)
        (METHOD = BASIC)
        (RETRIES = 180)
        (DELAY = 5)
      )
    )
  )

DEVDB =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip)(PORT = 1521))
    (ADDRESS = (PROTOCOL = TCP)(HOST = rac2-vip)(PORT = 1521))
    (LOAD_BALANCE = yes)
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = devdb)
    )
  )

EXTPROC_CONNECTION_DATA =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC0))
    )
    (CONNECT_DATA =
      (SID = PLSExtProc)
      (PRESENTATION = RO)
    )
  )

...
Mounting other filesystems:
     mount.ocfs2: Unable to access cluster service

Cannot initialize cluster mount.ocfs2:
    Unable to access cluster service Cannot initialize cluster [FAILED]
...

Before attempting to configure the on-boot properties:

• REMOVE the following lines in /etc/init.d/o2cb

  ### BEGIN INIT INFO
  # Provides: o2cb
  # Required-Start:
  # Should-Start:
  # Required-Stop:
  # Default-Start: 2 3 5
  # Default-Stop:
  # Description: Load O2CB cluster services at system boot.
  ### END INIT INFO

• Re-register the o2cb service.

  # chkconfig --del o2cb
  # chkconfig --add o2cb
  # chkconfig --list o2cb
  o2cb            0:off   1:off   2:on    3:on    4:on    5:on    6:off
  
  # ll /etc/rc3.d/*o2cb*
  lrwxrwxrwx  1 root root 14 Sep 29 11:56 /etc/rc3.d/S24o2cb -> ../init.d/o2cb

  The service should be S24o2cb in the default runlevel.

# /etc/init.d/o2cb offline ocfs2
# /etc/init.d/o2cb unload
# /etc/init.d/o2cb configure
Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot.  The current values will be shown in brackets ('[]').  Hitting
 without typing an answer will keep that current value.  Ctrl-C
will abort.

Load O2CB driver on boot (y/n) [n]: y
Cluster to start on boot (Enter "none" to clear) [ocfs2]: ocfs2
Writing O2CB configuration: OK
Loading module "configfs": OK
Mounting configfs filesystem at /config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting cluster ocfs2: OK

OCFS2 - Adjusting the O2CB Heartbeat Threshold

After looking through the trace files for OCFS2, it was apparent that access to the voting disk was too slow (exceeding the O2CB heartbeat threshold) and causing the Oracle Clusterware software (and the node) to crash. On the console would be a message similar to the following:

...
Index 0: took 0 ms to do submit_bio for read
Index 1: took 3 ms to do waiting for read completion
Index 2: took 0 ms to do bio alloc write
Index 3: took 0 ms to do bio add page write
Index 4: took 0 ms to do submit_bio for write
Index 5: took 0 ms to do checking slots
Index 6: took 4 ms to do waiting for write completion
Index 7: took 1993 ms to do msleep
Index 8: took 0 ms to do allocating bios for read
Index 9: took 0 ms to do bio alloc read
Index 10: took 0 ms to do bio add page read
Index 11: took 0 ms to do submit_bio for read
Index 12: took 10006 ms to do waiting for read completion
(13,3):o2hb_stop_all_regions:1888 ERROR: stopping heartbeat on all active regions.
Kernel panic - not syncing: ocfs2 is very sorry to be fencing this system by panicing

The solution I used was to increase the O2CB heartbeat threshold from its default value of 7, to 601. Some setups may require an even higher setting. This is a configurable parameter that is used to compute the time it takes for a node to "fence" itself. During the installation and configuration of OCFS2, we adjusted this value in the section "Configure O2CB to Start on Boot and Adjust O2CB Heartbeat Threshold". If you encounter a kernel panic from OCFS2 and need to increase the heartbeat threshold, use the same procedures described in the section "Configure O2CB to Start on Boot and Adjust O2CB Heartbeat Threshold". If you are using an earlier version of OCFS2 tools (prior to ocfs2-tools release 1.2.2-1), the following describes how to manually adjust the O2CB heartbeat threshold.

First, let's see how to determine what the O2CB heartbeat threshold is currently set to. This can be done by querying the /proc file system as follows:

# cat /proc/fs/ocfs2_nodemanager/hb_dead_threshold
7

[fence time in seconds] = (O2CB_HEARTBEAT_THRESHOLD - 1) * 2

(7 - 1) * 2 = 12 seconds

(601 - 1) * 2 = 1,200 seconds

Let's see now how to manually increase the O2CB heartbeat threshold from 7 to 601. This task will need to be performed on all Oracle RAC nodes in the cluster. We first need to modify the file /etc/sysconfig/o2cb and set O2CB_HEARTBEAT_THRESHOLD to 601:

/etc/sysconfig/o2cb

# O2CB_ENABELED: 'true' means to load the driver on boot. O2CB_ENABLED=true # O2CB_BOOTCLUSTER: If not empty, the name of a cluster to start. O2CB_BOOTCLUSTER=ocfs2 # O2CB_HEARTBEAT_THRESHOLD: Iterations before a node is considered dead. O2CB_HEARTBEAT_THRESHOLD=601

After modifying the file /etc/sysconfig/o2cb, we need to alter the o2cb configuration. Again, this should be performed on all Oracle RAC nodes in the cluster.

# umount /ocfs/
# /etc/init.d/o2cb unload
# /etc/init.d/o2cb configure

Load O2CB driver on boot (y/n) [y]: y
Cluster to start on boot (Enter "none" to clear) [ocfs2]: ocfs2
Writing O2CB configuration: OK
Loading module "configfs": OK
Mounting configfs filesystem at /config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting cluster ocfs2: OK

# cat /proc/fs/ocfs2_nodemanager/hb_dead_threshold
601

It is important to note that the value of 601 I used for the O2CB heartbeat threshold will not work for all configurations. In some cases, the O2CB heartbeat threshold value had to be increased to as high as 901 in order to prevent OCFS2 from panicking the kernel.

Clean remove Oracle Clusterware (CRS) 10GR2 from a RHEL4
/etc/init.d/init.evmd stop
/etc/init.d/init.evmd disable
/etc/init.d/init.cssd stop
/etc/init.d/init.cssd disable
/etc/init.d/init.crsd stop
/etc/init.d/init.crsd disable
/etc/init.d/init.crs stop
/etc/init.d/init.crs disable
rm -rf /etc/oracle /etc/oraInst.loc /etc/oratab
rm -rf /etc/init.d/init.crsd /etc/init.d/init.crs /etc/init.d/init.cssd /etc/init.d/init.evmd
rm -rf /etc/rc2.d/K96init.crs /etc/rc2.d/S96init.crs etc/rc3.d/K96init.crs /etc/rc3.d/S96init.crs /etc/rc4.d/K96init.crs /etc/rc4.d/S96init.crs /etc/rc5.d/K96init.crs /etc/rc5.d/S96init.crs /etc/rc.d/rc0.d/K96init.crs /etc/rc.d/rc1.d/K96init.crs /etc/rc.d/rc6.d/K96init.crs /etc/rc.d/rc4.d/K96init.crs
cp /etc/inittab.orig /etc/inittab
rm -rf /etc/inittab.crs /etc/inittab.no_crs
rm -rf /tmp/*
rm -rf /tmp/.oracle
rm -rf /usr/local/bin/dbhome /usr/local/bin/oraenv /usr/local/bin/coraenv
rm -rf /var/tmp/.oracle
rm -rf /u01/app/oracle/product/10.2.0/crs_1/*
echo
echo "Remove on one Node the Shared Devices"
echo "rm -rf /u03/oracrs/*"
echo
fi

15. Improving VM and Oracle

1. Obtain a baseline test for relative comparisons
2. On VM host, exclude VM clients from active, online virus scans
3. Remove Windows Indexing Service - because don’t need fast file searches
4. Remove other extraneous Windows services
5. Change VM host registry settings to improve file system IO for databases
6. Optimize the VM host configuration and options
7. Optimize the VM client OS configuration and options for Oracle database
8. Remove other extraneous VM client OS services or daemons
9. Change VM client file system settings to improve IO performance for databases
10. Adjust VM client file system block size to more closely match Oracle block size

I’m going to explain what choices #5 and #9 are. As a reminder:
   * #5 = Change VM host registry settings to improve file system IO for databases
   * #9 = Change VM client file system settings to improve IO performance for databases
In reality, this is one and the same improvement simply being applied to two different virtualization levels: the host and each client. Thus I’m simply going to present how to accomplish this technique for both Windows and Linux, and then leave it to the reader to make sure to apply it properly across all their various virtualization layers.

Both the Windows NTFS and Linux ext2/3 file systems maintain multiple "meta-data" information related to file access - such as date created, last time updated, etc. So an IO request might actually generate multiple physical IO’s - one for the data file and one or more for updating the related meta-data. In the case of the VMware server, we really don’t care to keep OS detailed file system information about the hosted clients’ data file access - it’s simply neither useful nor critical (unless doing snapshots). And at the Oracle database level, we know Oracle accesses its files every so many seconds - so why pay to update the meta-data with that information. Oracle keeps its own timestamps (i.e. SCN) in the file headers.

For Windows, we simply adjust the following registry entry:
   HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\FileSystem\NtfsDisableLastAccessUpdate = 1

For Linux, there are several ways to accomplish the same result. We can set the attribute for the individual Oracle files as follows:
   chattr +A file_name
Or we can do it for an entire directory:
   chattr -R +A directory_name
However the best method (because it automatically handles any file additions) is to edit the /etc/fstab and add the NOATIME attribute:
/dev/sda6     /                 ext3      defaults,noatime   1 1
/dev/sda1    /boot            ext2      defaults,noatime   1 2
/dev/cdrom   /mnt/cdrom   iso9660  noauto,owner,ro   0 0
/dev/sda5    swap            swap     defaults               0 0