Building the 10g rac Database

Once you have successfully installed Oracle 10g binaries and CRS. Its time to build the database. This is not a complicated task for a DBA who has built oracle instances and databases. There are few RCA specific parameters that need to be added to the init.ora file. I'll not cover this topic in detail, but will provide enough information to build a new database.
1) The first task is to create the initialization file. I'll list the init parameters that are requuired for RAC.
db_name =
instance_name =
instance_number = 1
cluster_database_instances = 2local_listener = '(address=(protocol=tcp)(host=hostname)(port=1521))
cluster_database = true
thread = 1
2) I use DBCA to generate scripts to build the database. If you already have the scripts to build the database, you could use that as well.
3) Make sure the "cluster_database" parameter is set to "false" before building the database else the create database command will fail.
4) Create the database as you would create a non-RAC database.
5) Make sure you execute the script "$ORACLE_HOME/rdbms/admin/catclust.sql" after executing the create catalog scripts.
6) The next job is to create additional undo segments for other instances on the RAC cluster.Please add the corresponding undo tablespace name in the respective init.ora for instances.
create undo tablespace UNDO02 datafile size 10240m reuse autoextend off;
7) Create additional redo thread for other instances in the RAC cluster.
alter database add logfile thread 2
group 21 (‘//RD21.rdo’) size
…
1024m reuse;
alter database enable thread 2;
8) Shutdown the first instance.
sqlplus > shutdown immediate
9) Edit the init file and set "cluster_database to true"
startup the first instance.
10) Copy the initfile from the primary node and create initfile for second instance.
Make sure you edited the following parameters.
Thread = 2
instance_name =
instance_number = 2
local_listener = '(address=(protocol=tcp)(host=hostname)(port=1521))'
undo_tablespace = UNDO02
11) startup the second instance.
sqlplus> startup
12) verify both the databases are up and running from any node.
sqlplus> select * from gv$instance;
The query result should dsiplay all the instances you created for the RAC database.
13)Add database to CRS using "srvctl" commands.
$srvctl add database -d -o $ORACLE_HOME -y manual
$srvctl add instance -d -i -n
$srvctl add instance -d -i -n
14) if everything goes well, the crs_stat -t command should list the database and instances.
$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....re2.gsd application ONLINE ONLINE nodeA
ora....re2.ons application ONLINE ONLINE nodeA
ora....re2.vip application ONLINE ONLINE nodeA
ora.fvdev1.db application ONLINE ONLINE nodeA
ora....11.inst application ONLINE ONLINE nodeA
ora....12.inst application ONLINE ONLINE nodeB
ora....b02.gsd application ONLINE ONLINE nodeB
ora....b02.ons application ONLINE ONLINE nodeB
ora....b02.vip application ONLINE ONLINE nodeB

Next step is creating Listeners for RAC
--------------------------------------
1) Set the TNS_ADMIN variable in CRS.
$srvctl setenv nodeapps -n -t TNS_ADMIN=
$srvctl setenv nodeapps -n -t TNS_ADMIN=
2) Start netca from any one node after setting the "DISPLAY" variable.
$netca
3) select Cluster configuration and hit next.

4) select both the nodes to configure.

5) select listener configuration and add listener.

6) select the default listener name.

7) Select protocol "tcp" and port 1521 on the next screen.

8)sample content of the listener.ora file.

SID_LIST_LISTENER_name =
(SID_LIST =
(SID_DESC =
(SID_NAME = PLSExtProc)
(ORACLE_HOME = /u01/app/oracle/product/10.2.0)
(PROGRAM = extproc)
)
)

LISTENER_name =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = hostname)(PORT = 1521)(IP = FIRST))
(ADDRESS = (PROTOCOL = TCP)(HOST = xx.xx.xx.xx)(PORT = 1521)(IP = FIRST))
)
)

9) set up client side TAF by adding TAF tnsnames entry into tnsnames.ora file of the Oracle clients.

e.g.

name =
(DESCRIPTION =
(load_balance = off) --> you can set it to on if you choose
(failover = on)
(ADDRESS = (PROTOCOL = TCP)(HOST = nodea)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = nodeb)(PORT = 1521))
(CONNECT_DATA =

(SERVICE_NAME = db_name)
(failover_mode =
(type = select)
(method = basic)
(retries = 5)
(delay = 1)
)
)
)

Validation

----------

Try the following commands.

1) crs_stop -all

2) crs_start -all

3)crs_stat -t

4)srvctl start/stop database –d $DB_NAME

5)srvctl start/stop instance –d $DB_NAME –i $INSTANCE_NAME

6) srvctl start/stop nodeapps –n $node

Back up the vote disk on both nodes. This step needs to be run any time after a new node is added or an existing node is removed.

$ cp /u01/app/oracle/dbdata/vote/vote_file /u01/app/oracle/cluster/crs/backup/crs/vote_file.`date +%y%m%d`

$verify all components are ONLINE using "crs_stat -t"

Name Type Target State Host

------------------------------------------------------------

ora....E2.lsnr application ONLINE ONLINE nodeA

ora....re2.gsd application ONLINE ONLINE nodeA

ora....re2.ons application ONLINE ONLINE nodeA

ora....re2.vip application ONLINE ONLINE nodeA

ora.fvdev1.db application ONLINE ONLINE nodeA

ora....11.inst application ONLINE ONLINE nodeA

ora....12.inst application ONLINE ONLINE nodeB

ora....02.lsnr application ONLINE ONLINE nodeB

ora....b02.gsd application ONLINE ONLINE nodeB

ora....b02.ons application ONLINE ONLINE nodeB

ora....b02.vip application ONLINE ONLINE nodeB

Patching RDBMS binaries for Oracle 10g R2

Patching RDBMS binaries is pretty straight forward. I'll not be pasting any screen shots, but explain the steps required.

1) DBA sets the Oracle home to RDBMS home directory.

2) Execute "runInstaller" from staging area.

3) Validate the Oracle_home and hit next.

4) The installer automatically picks the cluster installation, if "/etc/oraInst.loc or /var/opt/oracle/oraInst.loc (in solaris) " is pointing to CRS_HOME.
hit next

5) Verify the summary and hit Install.

6) Excute "root.sh" on all nodes in the cluster and then exit OUI.

Installing 10G R2 binaries in Oracle RAC environment.

Once you have successfully installed CRS, The Oracle 10gR2 binary installation is relatively easy.
1) DBA sets the ORACLE_HOME for the RDBMS binary and the verify the environment variables.
$> . oarenv
$> env grep -i oracle
2) Execute "runInstaller" from the staging area and select the custom installation.

3) Specify the Oracle_home details and hit next

4) specify cluster installation and check all nodes on the cluster.

5) Specify the required components for installation.

6) Make sure that the prerequisite checks have no errors.

7) Provide the "dba" group name.

8) Select "Install software only "(I prefer to build the database later).

9) Review the summary screen and hit continue.

10) execute "root.sh" on both nodes from the location displayed on the GUI.

$rootsh=/path/to/root.sh

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/10.2.0

Enter the full pathname of the local bin directory: [/usr/local/bin]: ..

Entries will be added to the /var/opt/oracle/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.

11) Hit O.K. and exit the GUI.

Oracle CRS 10.2.0.4 patching..

It has been a while, I have posted something on my blog. I hate my blog to look like a ghost town. I'll discuss the Oracle 10.2.0.4 patching proceess here.

I used to install Oracle 10g r2 base binaries right after installing Oracle CRS. I was hit by a OCR corruption issue which created lot of core dumps and was forced to rebuilt my OCR files. The avoid any such issue, I would suggest Patch oracle CRS first and the install RDBMS software.

1) The first step to apply 10.2.0.4 patch to CRS is to shutdwon CRS on both nodes.

Usually the "root" user will have the privileges to shitdown the CRS. The SA or root user can shutdown crs using the command

$> crsctl stop crs or

$/etc/init.d/init.crs stop

Make sure CRS is shut down.

$ps -ef grep crs

root 968 1 0 12:11:45 ? 0:00 /bin/sh /etc/init.d/init.crsd run

2) Make sure you change the ownership of "VIPCA" back to "oracle" from "root".

3) Set the environment varaiable for "crs".

$> ./.oraenv crs

check the oracle_home variable is to CRS_HOME.

env grep ORACLE_HOME

4) cd to the 10.2.0.4 patch set directory and execute "runInstaller". The crs and rdbms use the same 10.20.4 patset(there is no seperate 10204 patch set for CRS).

5) DBA confirms the CRSHOME and hit next

6) DBA confirms the node names in cluster and hit next

7) Validate the summary page and start installation.

8) At the end of the installation, Pl execute the root102.sh on all the nodes as instructed in the GUI.

9)log in as root and execute on all nodes one by one. you canignore the warnings below.

$/u01/app/oracle/cluster/crs/install/root102.shCreating pre-patch directory for saving pre-patch clusterware filesCompleted patching clusterware files to /u01/app/oracle/cluster/crsRelinking some shared libraries.ar: writing /u01/app/oracle/cluster/crs/lib/libn10.aar: writing /u01/app/oracle/cluster/crs/lib32/libn10.aar: writing /u01/app/oracle/cluster/crs/lib/libn10.aRelinking of patched files is complete.WARNING: directory '/u01/app/oracle/cluster' is not owned by rootWARNING: directory '/u01/app/oracle' is not owned by rootPreparing to recopy patched init and RC scripts.Recopying init and RC scripts.Startup will be queued to init within 30 seconds.Starting up the CRS daemons.Waiting for the patched CRS daemons to start. This may take a while on some systems..10204 patch successfully applied.clscfg: EXISTING configuration version 3 detected.clscfg: version 4 is 10G Release 2.Successfully accumulated necessary OCR keys.Using ports: CSS=32845 CRS=45632 EVMC=43567 and EVMR=34834.node : node 0: nodeA nodea-priv1 nodeBCreating OCR keys for user 'root', privgrp 'dba'..Operation successful.clscfg -upgrade completed successfully

10) Check CRS by issueing crs_stat -t and exit OUI.

$>crs_stat -t

Name Type Target State Host

-----------------------------

ora.node1.gsd application ONLINE ONLINE node1

ora.node1.ons application ONLINE ONLINE node1

ora.node1.vip application ONLINE ONLINE node1

ora.node2.gsd application ONLINE ONLINE node2

ora.node2.ons application ONLINE ONLINE node2

ora.node2.vip application ONLINE ONLINE node2

Oracle RAC Install on AIX using Veritas Clusterware.

Oracle RAC Install on AIX using Veritas Clusterware.

This document highlights the steps to install and new Oracle RAC binary on AIX cluster system using Veritas Clusterware without ASM. I’ll be discussing only the CRS/ORACLE installation and database setup. There is a lot of work that needs to be done prior to installing the Oracle software,(ie os, hardware an network setup) which we’ll not be covering here.

Assumptions: - The hardware is ready and the OS, network and Heart beat are setup properly. Veritas software is loaded and the storage disks are configured (A sample layout of the storage I have used for RAC is shown below (Table 1.00)

Table 1.00
FILE SYSTEM NAME SIZE COMMENT

/u0/app/oracle/cluster/crs 12GB CRS home and Logs
/u02/app/oracle/product 16GB ORACLE_HOME
/u03/app/oracle/admin 12GB Oracle dump directory
/uo4/misc 20GB Work space
/u05h/app/oracle/arch 20GB Archive space
/u06h/app/oracle/ocr 12GB For OCR
/u07h/app/oracle/vote 12GB For Vote disk
/u08h/app/oracle/oradata/system01 10GB For system datafiles
/u09h/app/oracle/oradata/system02 10GB For system datafiles
/u10h/app/oracle/oradata/system03 10GB For system data files.
/u11h/,,,….. As needed For apps data files …… As needed “
Uxxh As needed “

Note : - the letter “h” denotes that the filesystem is global (clustered). I have used local file system for Oracle and CRS binaries.These sizes are sample sizes need not be suitable for your environment.

The Oracle and CRS binaries were installed using the “oracle:dba” account. No two separate accounts were used for “oracle” and “CRS” installation. The oracle and CRS binaries were installed on non-global file systems.

Installation Process.

Before you begin to perform the installation.
· Copy the required binaries “the oracle clusterware for 10g”(The s/w can be downloaded from http://www.oracle.com/technology/software/products/database/oracle10g/htdocs/10201aixsoft.html) · Make sure that the global file systems are accessible from both the nodes.
· Make sure you could “ssh’ to both the boxes. (From node1 to node2 and viceversa).
· Make sure VCS is online and it good state.
· Verify the hearbeat connection.
· Make sure your environment is set to CRS_HOME.
ORACLE_HOME=/u01/app/oracle/crs/bin ; export ORACLE_HOME ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE ORACLE_SID=crs ; export ORACLE_SID umask 022 EXPORT PATH=$ORACLE_HOME/bin;$PATH EXPORT LD_LIBRARY_PATH=$ORACLE_HOME/lib
· Make sure you have set the oracle recommended kernel parameters.
· Verify the /etc/hosts file on all the nodes. It should have a public IP address for both the nodes,

private IP address for both nodes and a VIP (virtual IP ) configured for both nodes.

$> more /etc/hosts
# Public
xxx.xxx.xx.xxx node1.localdomain node1
xxx.xxx.xx.xxx node2.localdomain node2

#Private
xxx.xxx.xx.xxx node1-pvt.localdomain node1-pvt
xxx.xxx.xx.xxx node2-pvt.localdomain node2-pvt

#Virtual
xxx.xxx.xx.xxx node1-vrt.localdomain node1-vrt
xxx.xxx.xx.xxx node2-vrt.localdomain node2-vrt

CRS Installation
From a DBA perspective CRS installation can be complex, if the hardware is not not configured properly. It’s better to have a good understanding about the concepts before proceeding. Understand what the VIPS are for how RAC is using them. I would recommend reading the oracle RAC installation documents and get a thorough understanding of the hardware requirements and setup details.
1. Start the media installation for CRS.

./runInstaller

Login as root on another screen and run “rootpre.sh”. Once finished continue here by entering “y” at the prompt.


2. You’ll see the Oracle installer welcome page. Hit next


3. Enter your inventory path and group name.


4. Enter the crs_home name and path to install the crs. Hit next. You might see a warning saying that “The directory “ is not empty, in most cases this is O.k. you can proceed.


5. Hit next. You will see a screen with the results of the pre-requisite checks. Check for any warning and proceed. The next screen you see might look something similar to this.

If you don’t see the node names in the screen, it shows a problem(pl see issues faced) . If this screen is populated like shown above, then you are good. The n/w address information most of the time will be auto generated. Pl edit the address and
change it according to your settings. Hit ok and then next.



6. Edit and change the n/w address if necessary.




7.In the next screen the DBA checks the external redundancy box and enter global file name allocated for Oracle cluster registry (OCR). I have decided to go with external redundancy.


8.Next screen prompts you to enter the Voting Disk Location. Check the external redundancy and enter the voting disk location


9. The next screen shows you the summary. Review the summary and hit Install. The CRS installation begins and

10. The installer stops and asks the following scripts to be run. In some cases I have seen it asks you to run only root.sh. That’s normal. Don’t hit ok. Continue from step 11.


11. Execute the scripts and hit OK


12. Once the scripts are successfully executed. Log in as root on a separate window and start VIPCA( Virtual IP config assistant).


13.

14. select the public interface from the list


15. Enter the virtual alias. Hit next review the summary and hit Finish.Exit when done


17. If everything goes well go back to step 10 and hit OK at CRS GUI install window. The installer does a final configuration check and exits.

18. Look at the status of CRS,

I would check for the CRS processes running on the box.
ps –ef grep –i crs
You should see: - crsd.bin, ocssd.bin, evmd.bin, emvlogger.bin

You can also execute the command crs_stat –t . which should see the “gsd”, “ons” and “vip” processes.

oracle@> crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.node1.gsd application ONLINE ONLINE node1
ora.node1.ons application ONLINE ONLINE node1
ora.node1.vip application ONLINE ONLINE node1
ora.node2.gsd application ONLINE ONLINE node2
ora.node2.ons application ONLINE ONLINE node2
ora.node2.vip application ONLINE ONLINE node2

Issues Faced

• Step 5 the n/w addresses were not populated.

This shows that “oracle” user which performs the installation doesn’t have read privs on some system files. See whether oracle user has read permissions on /etc/llt* files.

• The specified nodes are not clusterable on step 5

Look in /etc/hosts file and make sure that the Ip address and names are entered in the same format as shown in the last bullet point of installation process and you IP addresses are right.

Conclusion

For first timer it’s slightly puzzling and as you do more you get to into the groove well. As I said in the beginning its better to have a good understanding of the concept before installing CRS. I’ll be discussing the Oracle binary installation and Patching process next.



I'll be discussing more of Oracle RDBMS install and pacthing process in the near future stay tuned.

Long Delay

Its been a while I have posted on this Blog. I am currently working on setting up a 10.2.0.4 RAC on solaris eneviornment on a Veritas Cluster environment.I'll get the details on teh set up soon. I promise you dont have to wait too long. I am also working on setting up a Oracle Internet Directory for Names Resolution. Lot of Things are on the way ...

ORACLE RAC

Let me start off with Oracle RAC (Real application cluster). I am sure if you are an Oracle professional, you must have some experience or knowledge about Oracle RAC technology. Oracle RAC is more refined or matured product of its predecessor “Oracle parallel server” which started off in the version 7 of oracle. The parallel server technology at that time is not used by a whole lot people and had some real drawbacks.

Oracle parallel server has improved a lot since it started off and is getting better with each version. In oracle 8i version of the parallel server, Oracle has introduced a new feature called “CACHE FUSION”, which I feel is significant achievement/turning point. This feature helps the other participating instances in the parallel server environment to read/share the SGA (system global area) of the other instances. In previous versions of RAC oracle has to write the data to disk first, before it becomes available for other participating instances.
In Oracle 9i this product officially received the name Real Application Cluster (RAC). In my experience I would consider Oracle 9i version of RAC as a real parallel database with many features. The installation and administration process has simplified a lot. The evidence is more and more people started using oracle 9i RAC database. It was difficult to get an Oracle professional in Oracle 8i days who posses oracle parallel server experience. It’s not the case after oracle 9i RAC was introduced.

Oracle has introduced many new and revolutionary changes to the RAC product in the 10g version. The most significant is the introduction of CRS (Cluster Ready Service) which is Oracle’s own cluster ware. The thing I like with CRS is that, it can work with other third party clustering tools, if you don’t prefer to use CRS as clustering software. Oracle has also provided many tools for administration ease such as “srvctl” “crs_stat” etc. I think Oracle still has a long way to go. I was stuck with a simple issue after the RAC build and it took almost 2 months to resolve, there is no good utilities to debug some of the common issues even in 10g.

I haven’t really got a chance to put my hands on the Oracle 11g RAC, but sure will get to it soon. I have gone through some of the Oracle 11g manuals and have some book knowledge. Unless I try them out, I don’t feel comfortable taking about it.

More and More people started using RAC databases and the demand will increase over the time. If you are already a DBA with oracle experience, it doesn’t take much time to get acquainted with the RAC system. If you are in search of an Oracle DBA job, experience in managing RAC databases is an added advantage.

I'll be discussing in detail more on the Oracle RAC installation and setup soon. Stay tuned.