Oracle RAC 10g and 11g on CentOS 5.2

ORACLE RAC LAB BUILD

The purpose of the RAC Lab is to:

a) build a similar system to the work RAC systems and to go through the build installation to identify configuration and installation issues before attempting the same work on the systems at work.

b) become familiar with RAC configuration and installation

c) create a sandbox platform to experiment with different configurations and functional testing

LAB Equipment:

Servers:
2 Dual AMD Opteron 64bit w/4Gb RAM and 2 gigabit ethernet ports
1 120Gb internal HD

Operating system:
CentOS 5.2 (Red Hat 5.2 binary equiv)

Oracle Software:
Cluster file system OCFS2 1.4.1
ASMLib 2.0
11g Oracle Clusterware
11g RAC ASM
11g Database using 11g RAC ASM
10g Database using 11g RAC ASM

Shared Storage:
Openfiler 2.3
Installed on AMD PC with 4Gb RAM single 640Gb HD and single gigabit
ethernet port

Network Equipment:
1 8 port gigabit switch (Private Interconnect and Storage Connection)
1 4 port 100mb/s hub (Public)

Staging server:
Apple Macbook

SIGNIFICANT DIFFERENCES BETWEEN LAB SETUP AND WORK SERVERS
=======================================================================================
ITEM LAB WORK
-----------------------------------------------------------------------------------------------------------------------------
Hardware 2 Rackable Systems 2 HP DL585 G5 s
Dual AMD Opteron 4 Quad Core AMD Opterons
single core CPU s w/128Gb RAM
w/4Gb RAM & Dual 6 Gigabit Ethernet w/4
Gigabit Ethernet bonded in pairs for private
and public networks 3 HBA s for SAN connections

Disk iSCSI disks from EMC SAN Disks Multipathed
dedicated PC with & connected through HBA s
openfiler 2.3
connected to the
RAC nodes via the
private network

OS CentOS 5.2 Red Hat 5.2

Security No special add ons Keon ESM

Monitoring No special add ons Tivoli netsnmp/snmp-DC
BMC Perform&Predict

Backup No special add ons Veritas Netbackup

DB Monitoring Local DB Console Oracle GridControl (Centralized)

INSTALLATION ACTIONS
=======================================================================================
1)INSTALL OS ON EACH NODE

choose CentOS 5.2 x86_64
select base install
when it reboots for the first time login:
disable firewall and SELinux
skip creating a user
log in as root
Do NOT run automatic updates when complete!!!

2)SETUP NETWORK

Open network admin utility and set up Network

Set hostname
disable DHCP

port eth0 = 10.0.1.6X
port eth1 = 10.0.2.6X

restart network services

Open terminal and edit /etc/hosts

Remove hostname from 127.0.0.1

add the following:

10.0.1.69 racbaby1
10.0.1.68 racbaby2
10.0.2.69 racbaby1-priv
10.0.2.68 racbaby2-priv
10.0.1.169 racbaby1-vip
10.0.1.168 racbaby2-vip
10.0.2.66 openfiler1

3)MODIFY KERNEL AND SETTINGS

use GUI "services" utility to disable updater card reader bluetooth etc

use GUI "Add Software" tool to install these items listed under "development"
development tools
development libraries
java development
legacy systems development

In a terminal window install the following additional packages:

yum install libaio-devel
yum install sysstat
yum install elfutils-libelf-devel
yum install iscsi-initiator-utils.x86_64
yum install lsscsi
yum install xorg-x11-deprecated-libs
yum install libXp.so.6

Set parameters using these commands:

sysctl -w net.core.rmem_default=4194304
sysctl -w net.core.wmem_default=262144
sysctl -w net.core.rmem_max=4194304
sysctl -w net.core.wmem_max=262144
echo "# Default setting in bytes of the socket receive buffer" >>/etc/sysctl.conf
echo "net.core.rmem_default= 4194304" >>/etc/sysctl.conf
echo "# Default setting in bytes of the socket send buffer" >>/etc/sysctl.conf
echo "net.core.wmem_default=262144" >>/etc/sysctl.conf
echo "# Maximum socket receive buffer size which may be set by using" >>/etc/sysctl.conf
echo "# the SO_RCVBUF socket option" >>/etc/sysctl.conf
echo "net.core.rmem_max= 4194304" >>/etc/sysctl.conf
echo "# Maximum socket send buffer size which may be set by using" >>/etc/sysctl.conf
echo "# the SO_SNDBUF socket option" >>/etc/sysctl.conf
echo "net.core.wmem_max=262144" >>/etc/sysctl.conf

sysctl -w kernel.shmmax=2147483648
echo "kernel.shmmax=2147483648" >> /etc/sysctl.conf
sysctl -w kernel.sem="250 32000 100 128"
echo "kernel.sem=250 32000 100 128" >> /etc/sysctl.conf
sysctl -w fs.file-max=65536
echo "fs.file-max=65536" >> /etc/sysctl.conf
sysctl -w net.ipv4.ip_local_port_range="1024 65000"
echo "net.ipv4.ip_local_port_range = 1024 65000" >> /etc/sysctl.conf

cat >> /etc/security/limits.conf <<EOF
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
EOF

cat >> /etc/pam.d/login <<EOF
session required /lib/security/pam_limits.so
EOF

cat >> /etc/profile <<EOF
if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi
EOF

4)CREATE ORACLE USER

groupadd -g 115 oinstall
groupadd -g 116 dba
useradd -m -u 175 -g oinstall -G dba -d /home/oracle -s /bin/bash -c "Oracle Software Owner" oracle
passwd oracle

5)CREATE SPACE FOR HOMES

mkdir -p /u01/app/oracle
chown -R oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/app/oracle
mkdir -p /u01/app/11.1.0
chown -R oracle:oinstall /u01/app/11.1.0
chmod -R 775 /u01/app/11.1.0

6)CREATE ISCSI DISKS

Use openfiler to set up disks and map luns to the two nodes. I m not going to
listout how I setup Openfiler as it doesn t pertain to the purpose of the LAB
build.

For simplicity I named the luns according to function so that when they appear on the nodes you ll see something like the following in /var/log/messages:

[root@racbaby2 log]# cat messages.1|grep openf|sort
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm10 10.0.2.66 3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm1 10.0.2.66 3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm2 10.0.2.66 3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm3 10.0.2.66 3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm4 10.0.2.66 3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm5 10.0.2.66 3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm6 10.0.2.66 3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm7 10.0.2.66 3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm8 10.0.2.66 3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:asm9 10.0.2.66 3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:ocfs 10.0.2.66 3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:ocr1 10.0.2.66 3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:ocr2 10.0.2.66 3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:spfile 10.0.2.66 3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:vote1 10.0.2.66 3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:vote2 10.0.2.66 3260]
Mar 4 09:47:15 racbaby2 iscsid: session [iqn.2006-01.com.openfiler:vote3 10.0.2.66 3260]

7)COPY OVER SOFTWARE FILES FROM STAGING SERVER

Hangcheck Timer
hangcheck-0.8.0.tar.gz

Oracle CLUSTER and DB Software packages:
linux.x64_11gR1_clusterware.zip
linux.x64_11gR1_database_1013.zip
10201_database_linux_x86_64.cpio.gz

Oracle OCFS RPM s
ocfs2-2.6.18-92.el5-1.4.1-1.el5.x86_64.rpm
ocfs2-tools-1.4.1-1.el5.x86_64.rpm
ocfs2-tools-debuginfo-1.4.1-1.el5.x86_64.rpm
ocfs2-tools-devel-1.4.1-1.el5.x86_64.rpm
ocfs2console-1.4.1-1.el5.x86_64.rpm

ASMLib RPM s
oracleasm-2.6.18-92.1.22.el5-2.0.5-1.el5.x86_64.rpm
oracleasm-2.6.18-92.el5-2.0.5-1.el5.x86_64.rpm
oracleasm-support-2.1.2-1.el5.x86_64.rpm
oracleasmlib-2.0.3-1.el5.x86_64.rpm

homemade env variable switcher script
oradb

Oracle patchsets:
p6810189_10204_Linux-x86-64.zip
p6890831_111070_Linux-x86-64.zip

8)SET UP ISCSI ON RAC NODES

as root on each node:

service iscsi restart
chkconfig --level 345 iscsi on
iscsiadm -m discovery -t sendtargets -p openfiler1
service iscsi restart

verify disks are there

fdisk -l

run fdisk on each device to create a primary partition

1z0-963-exams

Hmm it looks like your blog ate my first comment (it was super long) so I guess I'll just sum it up what I had written and say, I'm thoroughly enjoying your blog. I too am an aspiring blog blogger but I'm still new to everything. Do you have any suggestions for first-time blog writers? I'd definitely appreciate it.

Hmm it looks like your blog ate my first comment (it was super long) so I guess I'll just sum it up what I had written and say, I'm thoroughly enjoying your blog. I too am an aspiring blog blogger but I'm still new to everything. Do you have any suggestions for first-time blog writers? I'd definitely appreciate it.

Edmonton Tirecraft Auto Service

Thank you for your website post. Brown and I are already saving to buy a new e book on this topic and your writing has made people like us to save all of our money. Your ideas really answered all our concerns. In fact, a lot more than what we had known ahead of the time we came upon your excellent blog. My spouse and i no longer have doubts along with a troubled mind because you have attended to our own needs above. Thanks

Thank you for your website post. Brown and I are already saving to buy a new e book on this topic and your writing has made people like us to save all of our money. Your ideas really answered all our concerns. In fact, a lot more than what we had known ahead of the time we came upon your excellent blog. My spouse and i no longer have doubts along with a troubled mind because you have attended to our own needs above. Thanks

bolu escort

Please let me know if you're looking for a author for your site. You have some really great articles and I believe I would be a good asset. If you ever want to take some of the load off, I'd absolutely love to write some content for your blog in exchange for a link back to mine. Please send me an email if interested. Thank you!

Please let me know if you're looking for a author for your site. You have some really great articles and I believe I would be a good asset. If you ever want to take some of the load off, I'd absolutely love to write some content for your blog in exchange for a link back to mine. Please send me an email if interested. Thank you!

kuruyemiş makina

After looking at a handful of the blog articles on your site, I seriously appreciate your technique of writing a blog. I book marked it to my bookmark webpage list and will be checking back in the near future. Take a look at my web site as well and let me know how you feel.

After looking at a handful of the blog articles on your site, I seriously appreciate your technique of writing a blog. I book marked it to my bookmark webpage list and will be checking back in the near future. Take a look at my web site as well and let me know how you feel.

kredi notu öğrenme

all the time i used to read smaller content which also clear their motive, and that is also happening with this paragraph which I am reading here.

all the time i used to read smaller content which also clear their motive, and that is also happening with this paragraph which I am reading here.

denizli haber

I have been browsing online greater than three hours today, but I never found any fascinating article like yours. It is pretty worth sufficient for me. In my view, if all website owners and bloggers made good content as you did, the web will probably be a lot more helpful than ever before.

I have been browsing online greater than three hours today, but I never found any fascinating article like yours. It is pretty worth sufficient for me. In my view, if all website owners and bloggers made good content as you did, the web will probably be a lot more helpful than ever before.

samsung note 8 teknosa

Go Ƅig wіth the Galaxy Note8's 6.3" screen.

Go big witһ the Galaxy Note8's 6.3" screen.

i need a payday loan immediately

Right here is the right blog for everyone who would like to find out about this topic. You know a whole lot its almost tough to argue with you (not that I personally would want to…HaHa). You certainly put a brand new spin on a topic that's been written about for ages. Excellent stuff, just great!

Right here is the right blog for everyone who would like to find out about this topic. You know a whole lot its almost tough to argue with you (not that I personally would want to…HaHa). You certainly put a brand new spin on a topic that's been written about for ages. Excellent stuff, just great!

wow

Hmm is anyone else experiencing problems with the images on this blog loading? I'm trying to figure out if its a problem on my end or if it's the blog. Any feed-back would be greatly appreciated.

Hmm is anyone else experiencing problems with the images on this blog loading? I'm trying to figure out if its a problem on my end or if it's the blog. Any feed-back would be greatly appreciated.

diabetes miracle cure guide

I have learn some just right stuff here. Certainly price bookmarking for revisiting. I surprise how a lot effort you place to create this sort of excellent informative website.

I have learn some just right stuff here. Certainly price bookmarking for revisiting. I surprise how a lot effort you place to create this sort of excellent informative website.