Exadata: Initial Installation Validation

After our successful implementation of our Exadata Quarter Rack, we wanted to share our configuration details and scripts used to validate the installation. As you probably know, the Exadata Database Machine is a combination of compute nodes (database servers), storage cell nodes, a high-speed InfiniBand interconnect network, an embedded Cisco switch, a KVM switch, all contained in a purpose-built rack with redundant power supplies. In this blog post, I'll show actual information and details from our X2-2 Quarter Rack, built to the specifications in our configuration template.

Centroid Exadata X2-2 Quarter Rack

[root@cm01dbm01 ~]# uname -a
 

Linux cm01dbm01.centroid.com
2.6.18-194.3.1.0.4.el5 #1 SMP Sat Feb 19 03:38:37 EST 2011 x86_64
x86_64 x86_64 GNU/Linux
 

[root@cm01dbm01 ~]# cat /etc/redhat-release
 

Enterprise Linux Enterprise Linux Server release
5.5 (Carthage)
 

[root@cm01dbm01 ~]#

[root@cm01cel01 ~]# uname -a

Linux cm01cel01.centroid.com
2.6.18-194.3.1.0.4.el5 #1 SMP Sat Feb 19 03:38:37 EST 2011 x86_64
x86_64 x86_64 GNU/Linux

[root@cm01cel01 ~]# cat /etc/redhat-release

Enterprise Linux Enterprise Linux Server release
5.5 (Carthage)

[root@cm01cel01 ~]#

[root@cm01dbm01 ~]# imageinfo

Kernel version: 2.6.18-194.3.1.0.4.el5 #1 SMP
Sat Feb 19 03:38:37 EST 2011 x86_64

Image version: 11.2.2.2.2.110311

Image activated: 2011-05-04 12:41:40 -0400

Image status: success

System partition on device:
/dev/mapper/VGExaDb-LVDbSys1

[root@cm01cel01 ~]# imageinfo

Kernel version: 2.6.18-194.3.1.0.4.el5 #1 SMP
Sat Feb 19 03:38:37 EST 2011 x86_64

Cell version:
OSS_11.2.2.2.2_LINUX.X64_110311

Cell rpm version:
cell-11.2.2.2.2_LINUX.X64_110311-1

Active image version: 11.2.2.2.2.110311

Active image activated: 2011-05-04 12:31:56
-0400

Active image status: success

Active system partition on device: /dev/md6

Active software partition on device:
/dev/md8

In partition rollback: Impossible

Cell boot usb partition: /dev/sdac1

Cell boot usb version: 11.2.2.2.2.110311

Inactive image version: 11.2.2.2.0.101206.2

Inactive image activated: 2011-02-21 11:20:38
-0800

Inactive image status: success

Inactive system partition on device:
/dev/md5

Inactive software partition on device:
/dev/md7

Boot area has rollback archive for the version:
11.2.2.2.0.101206.2

Rollback to the inactive partitions:
Possible

[grid@cm01dbm01 ~]$ crsctl query crs
activeversion

Oracle Clusterware active version on the cluster
is [11.2.0.2.0]

[grid@cm01dbm01 ~]$

[oracle@cm01dbm01 ~]$ sqlplus /nolog

SQL*Plus: Release 11.2.0.2.0 Production on Thu
May 5 22:01:24 2011

Copyright (c) 1982, 2010, Oracle. All rights
reserved.

SQL>

[root@cm01cel01 ~]# cellcli -e list cell
detail|grep cellVersion

cellVersion: OSS_11.2.2.2.2_LINUX.X64_110311

[root@cm01cel01 ~]#

1 select name,value from v$asm_attribute

2* where name like 'compat%'

SQL> /

NAME VALUE

-------------------- --------------------

compatible.asm 11.2.0.2.0

compatible.rdbms 11.2.0.2

compatible.asm 11.2.0.2.0

compatible.rdbms 11.2.0.2

compatible.asm 11.2.0.2.0

compatible.rdbms 11.2.0.2

[oracle@cm01dbm01 ~]$ srvctl status nodeapps

VIP cm0101-vip is enabled

VIP cm0101-vip is running on node: cm01dbm01

VIP cm0102-vip is enabled

VIP cm0102-vip is running on node: cm01dbm02

Network is enabled

Network is running on node: cm01dbm01

Network is running on node: cm01dbm02

GSD is disabled

GSD is not running on node: cm01dbm01

GSD is not running on node: cm01dbm02

ONS is enabled

ONS daemon is running on node: cm01dbm01

ONS daemon is running on node: cm01dbm02

[oracle@cm01dbm01 ~]$

[oracle@cm01dbm01 ~]$ srvctl status asm

ASM is running on cm01dbm02,cm01dbm01

[oracle@cm01dbm01 ~]$

[oracle@cm01dbm01 ~]$ srvctl status database -d
dbm

Instance dbm1 is running on node cm01dbm01

Instance dbm2 is running on node cm01dbm02

[oracle@cm01dbm01 ~]$

[oracle@cm01dbm01 ~]$ srvctl status service -d
dbm

[oracle@cm01dbm01 ~]$

[grid@cm01dbm01 ~]$ oifcfg iflist -p

eth0 172.16.1.0 PRIVATE

eth3 172.16.100.0 PRIVATE

bondeth0 172.16.10.0 PRIVATE

bondib0 192.168.8.0 PRIVATE

bondib0 169.254.0.0 UNKNOWN

[grid@cm01dbm01 ~]$

[grid@cm01dbm01 admin]$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.2.0 -
Production on 05-MAY-2011 22:11:47

Copyright (c) 1991, 2010, Oracle. All rights
reserved.

Connecting to
(DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))

STATUS of the LISTENER

------------------------

Alias LISTENER

Version TNSLSNR for Linux: Version 11.2.0.2.0 -
Production

Start Date 04-MAY-2011 14:30:13

Uptime 1 days 7 hr. 41 min. 34 sec

Trace Level off

Security ON: Local OS Authentication

SNMP OFF

Listener Parameter File
/u01/app/11.2.0/grid/network/admin/listener.ora

Listener Log File
/u01/app/grid/diag/tnslsnr/cm01dbm01/listener/alert/log.xml

Listening Endpoints Summary...

(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))

(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=172.16.1.10)(PORT=1521)))

(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=172.16.10.12)(PORT=1521)))

Services Summary...

Service "+ASM" has 1 instance(s).

Instance "+ASM1", status READY, has 1 handler(s)
for this service...

Service "dbm" has 1 instance(s).

Instance "dbm1", status READY, has 1 handler(s)
for this service...

The command completed successfully

[grid@cm01dbm01 admin]$

Exadata Cell Details

The Exadata X2-2 Quarter Rack comes with three (3) Exadata Storage Cells. Ours are cm01cel01, cm01cel02, and cm01cel03. In this section, I'll be showing examples using dcli from the database tier nodes, but each command could be done on the storage cell using cellcli independently.

Currently, the cell_group file (which lists the cells) is located in /opt/oracle.SupportTools/onecommand/cell_groupand in /home/oracle/cell_group. We needed to setup SSH equivalency to the celladmin Linux account on the cell server nodes from both root and oracle on the compute nodes¹.

First, we set the celladmin password on all 3 cell server nodes. The initial password is "welcome". For Centroid, I've set the celladmin password the same as the root password on all nodes.

As oracle:

[oracle@cm01dbm01 ~]$ dcli -g cell_group -k

As root:

[root@cm01dbm01 ~]# dcli -g cell_group -k

celladmin@cm01cel02's password:

celladmin@cm01cel03's password:

celladmin@cm01cel01's password:

cm01cel01: ssh key added

cm01cel02: ssh key added

cm01cel03: ssh key added

[root@cm01dbm01 ~]#

Cell Disks

[oracle@cm01dbm01 ~]$ dcli -g cell_group cellcli
-e list celldisk

cm01cel01: CD_00_cm01cel01 normal

cm01cel01: CD_01_cm01cel01 normal

cm01cel01: CD_02_cm01cel01 normal

cm01cel01: CD_03_cm01cel01 normal

<< some output deleted >>

cm01cel01: FD_00_cm01cel01 normal

cm01cel01: FD_01_cm01cel01 normal

cm01cel01: FD_02_cm01cel01 normal

cm01cel01: FD_03_cm01cel01 normal

cm01cel01: FD_04_cm01cel01 normal

<< some output deleted>>

[oracle@cm01dbm01 ~]$

LUNs

[oracle@cm01dbm01 ~]$ dcli -g cell_group cellcli
-e list lun

cm01cel01: 0_0 0_0 normal

cm01cel01: 0_1 0_1 normal

cm01cel01: 0_2 0_2 normal

cm01cel01: 0_3 0_3 normal

cm01cel01: 0_4 0_4 normal

cm01cel01: 0_5 0_5 normal

cm01cel01: 0_6 0_6 normal

<< some output deleted>>

cm01cel03: 5_3 5_3 normal

[oracle@cm01dbm01 ~]$

Physical Disks

[oracle@cm01dbm01 ~]$ dcli -g cell_group cellcli
-e list physicaldisk

cm01cel01: 20:0 E1PJHG normal

cm01cel01: 20:1 E1R1DA normal

cm01cel01: 20:2 E1R743 normal

cm01cel01: 20:3 E1R3RR normal

<< some output deleted>>

cm01cel03: [1:0:1:0] 5080020000f3cb2FMOD1
normal

cm01cel03: [1:0:2:0] 5080020000f3cb2FMOD2
normal

cm01cel03: [1:0:3:0] 5080020000f3cb2FMOD3
normal

<< some output deleted>>

cm01cel03: [4:0:3:0] 5080020000f3f02FMOD3
normal

Cell Details

[oracle@cm01dbm01 ~]$ dcli -g cell_group cellcli
-e list cell detail

cm01cel01: name: cm01cel01

cm01cel01: bmcType: IPMI

cm01cel01: cellVersion:
OSS_11.2.2.2.2_LINUX.X64_110311

cm01cel01: cpuCount: 24

cm01cel01: fanCount: 12/12

cm01cel01: fanStatus: normal

cm01cel01: id: 1104FMM0MG

cm01cel01: interconnectCount: 3

cm01cel01: interconnect1: bondib0

cm01cel01: iormBoost: 0.0

cm01cel01: ipaddress1: 192.168.10.3/22

cm01cel01: kernelVersion:
2.6.18-194.3.1.0.4.el5

cm01cel01: locatorLEDStatus: off

cm01cel01: makeModel: SUN MICROSYSTEMS SUN FIRE
X4270 M2 SERVER SAS

cm01cel01: metricHistoryDays: 7

cm01cel01: offloadEfficiency: 4.5G

cm01cel01: powerCount: 2/2

cm01cel01: powerStatus: normal

cm01cel01: status: online

cm01cel01: temperatureReading: 24.0

cm01cel01: temperatureStatus: normal

cm01cel01: upTime: 0 days, 8:26

cm01cel01: cellsrvStatus: running

cm01cel01: msStatus: running

cm01cel01: rsStatus: running

cm01cel02: name: cm01cel02

cm01cel02: bmcType: IPMI

cm01cel02: cellVersion:
OSS_11.2.2.2.2_LINUX.X64_110311

cm01cel02: cpuCount: 24

cm01cel02: fanCount: 12/12

cm01cel02: fanStatus: normal

cm01cel02: id: 1104FMM0LG

cm01cel02: interconnectCount: 3

cm01cel02: interconnect1: bondib0

cm01cel02: iormBoost: 0.0

cm01cel02: ipaddress1: 192.168.10.4/22

cm01cel02: kernelVersion:
2.6.18-194.3.1.0.4.el5

cm01cel02: locatorLEDStatus: off

cm01cel02: makeModel: SUN MICROSYSTEMS SUN FIRE
X4270 M2 SERVER SAS

cm01cel02: metricHistoryDays: 7

cm01cel02: offloadEfficiency: 4.5G

cm01cel02: powerCount: 2/2

cm01cel02: powerStatus: normal

cm01cel02: status: online

cm01cel02: temperatureReading: 24.0

cm01cel02: temperatureStatus: normal

cm01cel02: upTime: 0 days, 8:26

cm01cel02: cellsrvStatus: running

cm01cel02: msStatus: running

cm01cel02: rsStatus: running

cm01cel03: name: cm01cel03

cm01cel03: bmcType: IPMI

cm01cel03: cellVersion:
OSS_11.2.2.2.2_LINUX.X64_110311

cm01cel03: cpuCount: 24

cm01cel03: fanCount: 12/12

cm01cel03: fanStatus: normal

cm01cel03: id: 1104FMM0M2

cm01cel03: interconnectCount: 3

cm01cel03: interconnect1: bondib0

cm01cel03: iormBoost: 0.0

cm01cel03: ipaddress1: 192.168.10.5/22

cm01cel03: kernelVersion:
2.6.18-194.3.1.0.4.el5

cm01cel03: locatorLEDStatus: off

cm01cel03: makeModel: SUN MICROSYSTEMS SUN FIRE
X4270 M2 SERVER SAS

cm01cel03: metricHistoryDays: 7

cm01cel03: offloadEfficiency: 4.5G

cm01cel03: powerCount: 2/2

cm01cel03: powerStatus: normal

cm01cel03: status: online

cm01cel03: temperatureReading: 25.0

cm01cel03: temperatureStatus: normal

cm01cel03: upTime: 0 days, 8:26

cm01cel03: cellsrvStatus: running

cm01cel03: msStatus: running

cm01cel03: rsStatus: running

[oracle@cm01dbm01 ~]$

Grid Disks

[oracle@cm01dbm01 ~]$ dcli -g cell_group cellcli
-e list griddisk

cm01cel01: DATA_CM01_CD_00_cm01cel01 active

cm01cel01: DATA_CM01_CD_01_cm01cel01 active

cm01cel01: DATA_CM01_CD_02_cm01cel01 active

<< some output deleted>>

cm01cel02: DBFS_DG_CD_04_cm01cel02 active

cm01cel02: DBFS_DG_CD_05_cm01cel02 active

<< some output deleted>>

cm01cel03: RECO_CM01_CD_11_cm01cel03 active

[oracle@cm01dbm01 ~]$

Flash Cache

[oracle@cm01dbm01 ~]$ dcli -g cell_group cellcli
-e list flashcache

cm01cel01: cm01cel01_FLASHCACHE normal

cm01cel02: cm01cel02_FLASHCACHE normal

cm01cel03: cm01cel03_FLASHCACHE normal

[oracle@cm01dbm01 ~]$

[oracle@cm01dbm01 ~]$ dcli -g cell_group cellcli
-e drop flashcache all

cm01cel01: Flash cache cm01cel01_FLASHCACHE
successfully dropped

cm01cel02: Flash cache cm01cel02_FLASHCACHE
successfully dropped

cm01cel03: Flash cache cm01cel03_FLASHCACHE
successfully dropped

[oracle@cm01dbm01 ~]$ dcli -g cell_group cellcli
-e create flashcache all

cm01cel01: Flash cache cm01cel01_FLASHCACHE
successfully created

cm01cel02: Flash cache cm01cel02_FLASHCACHE
successfully created

cm01cel03: Flash cache cm01cel03_FLASHCACHE
successfully created

[oracle@cm01dbm01 ~]$

Cluster, ASM, and Database

The overall Oracle topology of an Exadata Database Machine consists of:

• Compute nodes (database nodes) and cell servers (Exadata storage cells)

• An 11gR2 Grid Infrastructure home

• An 11gR2 RDBMS home using Oracle ASM on Exadata storage cells.

• The compute nodes are cm01dbm01 and cm01dbm02

At a high level, we've configured role-based security and used users grid for the GI home and oracle for the DB home. Our default database is called dbm, with instance dbm1 running on cm01dbm01 and dbm2 running on cm01dbm02.

Exadata Connection Specifics

The compute nodes, cm01dbm01 and cm01dbm02, communicate with the Exadata storage cells using details in:

/etc/oracle/cell/network-config/cellip.ora:

cell="192.168.10.3"

cell="192.168.10.4"

cell="192.168.10.5"

/etc/oracle/cell/network-config/cellinit.ora:

ipaddress1=192.168.10.1/22

Grid Infrastructure Home

The Unix account that owns the Grid Infrastructure is grid:

[grid@cm01dbm02 ~]$ whoami

grid

[grid@cm01dbm02 ~]$ id

uid=1000(grid) gid=1001(oinstall)
groups=1001(oinstall),1004(asmdba),1005(asmoper),1006(asmadmin)

[grid@cm01dbm02 ~]$

export CRS_HOME=/u01/app/11.2.0/grid

export ORACLE_HOME=$CRS_HOME

export PATH=$ORACLE_HOME/bin:$PATH

[grid@cm01dbm01 ~]$ crsctl check crs

CRS-4638: Oracle High Availability Services is
online

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is
online

CRS-4533: Event Manager is online

[grid@cm01dbm01 ~]$

[grid@cm01dbm02 ~]$ crsctl stat res -t

--------------------------------------------------------------------------------

NAME TARGET STATE SERVER STATE_DETAILS

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.DATA_CM01.dg

ONLINE ONLINE cm01dbm01

ONLINE ONLINE cm01dbm02

ora.DBFS_DG.dg

ONLINE ONLINE cm01dbm01

ONLINE ONLINE cm01dbm02

ora.LISTENER.lsnr

ONLINE ONLINE cm01dbm01

ONLINE ONLINE cm01dbm02

ora.RECO_CM01.dg

ONLINE ONLINE cm01dbm01

ONLINE ONLINE cm01dbm02

ora.asm

ONLINE ONLINE cm01dbm01

ONLINE ONLINE cm01dbm02

ora.gsd

OFFLINE OFFLINE cm01dbm01

OFFLINE OFFLINE cm01dbm02

ora.net1.network

ONLINE ONLINE cm01dbm01

ONLINE ONLINE cm01dbm02

ora.ons

ONLINE ONLINE cm01dbm01

ONLINE ONLINE cm01dbm02

ora.registry.acfs

ONLINE ONLINE cm01dbm01

ONLINE ONLINE cm01dbm02

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE cm01dbm01

ora.LISTENER_SCAN2.lsnr

1 ONLINE ONLINE cm01dbm02

ora.LISTENER_SCAN3.lsnr

1 ONLINE ONLINE cm01dbm02

ora.cm01dbm01.vip

1 ONLINE ONLINE cm01dbm01

ora.cm01dbm02.vip

1 ONLINE ONLINE cm01dbm02

ora.cvu

1 ONLINE ONLINE cm01dbm02

ora.dbm.db

1 ONLINE ONLINE cm01dbm01 Open

2 ONLINE ONLINE cm01dbm02 Open

ora.oc4j

1 ONLINE ONLINE cm01dbm02

ora.scan1.vip

1 ONLINE ONLINE cm01dbm01

ora.scan2.vip

1 ONLINE ONLINE cm01dbm02

ora.scan3.vip

1 ONLINE ONLINE cm01dbm02

[grid@cm01dbm02 ~]$

In Oracle 11.2 Grid Infrastructure, the listener runs out of the Grid Infrastructure (GI) home and can be queried with lsnrctl status and lsnrctl status LISTENER_SCAN[1|2]

Database Home

[oracle@cm01dbm02 ~]$ . oraenv

ORACLE_SID = [oracle] ? dbm

The Oracle base has been set to
/u01/app/oracle

[oracle@cm01dbm02 ~]$ echo $ORACLE_HOME

/u01/app/oracle/product/11.2.0/dbhome_1

[oracle@cm01dbm02 ~]$ id

uid=1001(oracle) gid=1001(oinstall)
groups=1001(oinstall),1002(dba),1003(racoper),1004(asmdba)

[oracle@cm01dbm02 ~]$

Our DBM Database

The Exadata installation installed a database called dbm with instances dbm1 and dbm2 installed on cm01dbm01 and cm01dbm02, respectively. This was sized at 100Gb for some reason, which seems very large. Since this database will be deleted at some point, we won't go into too much detail right now but I will show several key components. The SYS and SYSTEM passwords are "welcome"

SQL> select sum(bytes)/1024/1024/1024 from
v$datafile;

SUM(BYTES)/1024/1024/1024

-------------------------

72.8125

SQL>

1 select
tablespace_name,sum(bytes)/1024/1024/1024

2 from dba_data_files

3 group by tablespace_name

4* order by 1

SQL> /

TABLESPACE_NAME SUM(BYTES)/1024/1024/1024

------------------------------
-------------------------

SYSAUX 16

SYSTEM 16

UNDOTBS1 16

UNDOTBS2 16

USERS 8.8125

SQL> show sga

Total System Global Area 1.7103E+10 bytes

Fixed Size 2243608 bytes

Variable Size 2516583400 bytes

Database Buffers 1.4428E+10 bytes

Redo Buffers 155930624 bytes

SQL>

SQL> show parameter cell

NAME TYPE VALUE

------------------------------------ -----------
------------------------------

cell_offload_compaction string ADAPTIVE

cell_offload_decryption boolean TRUE

cell_offload_parameters string

cell_offload_plan_display string AUTO

cell_offload_processing boolean TRUE

SQL> show parameter cluster

NAME TYPE VALUE

------------------------------------ -----------
------------------------------

cluster_database boolean TRUE

cluster_database_instances integer 2

cluster_interconnects string 192.168.10.1

SQL>

RAC Details

Any Oracle RAC environment consists of the following:

• Shared storage, which in the case of Exadata is always on Oracle ASM:

SQL> select name from v$datafile;

NAME

--------------------------------------------------------------------------------

+DATA_CM01/dbm/datafile/system.261.750252107

+DATA_CM01/dbm/datafile/sysaux.262.750252119

+DATA_CM01/dbm/datafile/undotbs1.263.750252129

+DATA_CM01/dbm/datafile/undotbs2.265.750252149

+DATA_CM01/dbm/datafile/users.266.750252159

SQL>

ORACLE_SID = [grid] ? +ASM1

The Oracle base has been set to
/u01/app/grid

[grid@cm01dbm01 ~]$ asmcmd

ASMCMD> ls

DATA_CM01/

DBFS_DG/

RECO_CM01/

ASMCMD> cd DATA_CM01

ASMCMD> ls

DBM/

ASMCMD> cd DBM

ASMCMD> ls

CHANGETRACKING/

CONTROLFILE/

DATAFILE/

ONLINELOG/

PARAMETERFILE/

spfiledbm.ora

ASMCMD> cd DATAFILE

ASMCMD> ls

SYSAUX.262.750252119

SYSTEM.261.750252107

UNDOTBS1.263.750252129

UNDOTBS2.265.750252149

USERS.266.750252159

ASMCMD>

• Private interconnect (more details in the InfiniBand section below):

SQL> show parameter interconnect

NAME TYPE VALUE

------------------------------------ -----------
------------------------------

cluster_interconnects string 192.168.10.1

SQL>

• RAC node applications, which are essentially listeners, VIPs, ONS services:

[oracle@cm01dbm01 ~]$ srvctl status nodeapps

VIP cm0101-vip is enabled

VIP cm0101-vip is running on node: cm01dbm01

VIP cm0102-vip is enabled

VIP cm0102-vip is running on node: cm01dbm02

Network is enabled

Network is running on node: cm01dbm01

Network is running on node: cm01dbm02

GSD is disabled

GSD is not running on node: cm01dbm01

GSD is not running on node: cm01dbm02

ONS is enabled

ONS daemon is running on node: cm01dbm01

ONS daemon is running on node: cm01dbm02

[oracle@cm01dbm01 ~]$

• A single database and an instance on each node in the cluster:

[oracle@cm01dbm01 ~]$ srvctl status database -d
dbm

Instance dbm1 is running on node cm01dbm01

Instance dbm2 is running on node cm01dbm02

[oracle@cm01dbm01 ~]$

• A shared OCR file (Oracle Cluster Registry), which outlines which resources are enabled on each node:

[grid@cm01dbm01 ~]$ ocrcheck

Status of Oracle Cluster Registry is as follows
:

Version : 3

Total space (kbytes) : 262120

Used space (kbytes) : 2956

Available space (kbytes) : 259164

ID : 1833511320

Device/File Name : +DBFS_DG

Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to
non-privileged user

[grid@cm01dbm01 ~]$

[grid@cm01dbm01 ~]$ crsctl check crs

CRS-4638: Oracle High Availability Services is
online

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is
online

CRS-4533: Event Manager is online

[grid@cm01dbm01 ~]$

[grid@cm01dbm02 ~]$ crsctl stat res -t

--------------------------------------------------------------------------------

NAME TARGET STATE SERVER STATE_DETAILS

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.DATA_CM01.dg

ONLINE ONLINE cm01dbm01

ONLINE ONLINE cm01dbm02

ora.DBFS_DG.dg

ONLINE ONLINE cm01dbm01

ONLINE ONLINE cm01dbm02

ora.LISTENER.lsnr

ONLINE ONLINE cm01dbm01

ONLINE ONLINE cm01dbm02

ora.RECO_CM01.dg

ONLINE ONLINE cm01dbm01

ONLINE ONLINE cm01dbm02

ora.asm

ONLINE ONLINE cm01dbm01

ONLINE ONLINE cm01dbm02

ora.gsd

OFFLINE OFFLINE cm01dbm01

OFFLINE OFFLINE cm01dbm02

ora.net1.network

ONLINE ONLINE cm01dbm01

ONLINE ONLINE cm01dbm02

(etc ...)

• A shared set of Voting disks, which contain information about members of the cluster:

[grid@cm01dbm01 ~]$ crsctl query css
votedisk

## STATE File Universal Id File Name Disk
group

-- ----- ----------------- ---------
---------

1. ONLINE 748aa14664864feebf2a586b76c9710e
(o/192.168.10.3/DBFS_DG_CD_02_cm01cel01) [DBFS_DG]

2. ONLINE 0ccbfc8276364f57bfd95e8918d464b5
(o/192.168.10.4/DBFS_DG_CD_02_cm01cel02) [DBFS_DG]

3. ONLINE 6c4118053e6c4f55bf7b7713efc28c89
(o/192.168.10.5/DBFS_DG_CD_02_cm01cel03) [DBFS_DG]

Located 3 voting disk(s).

[grid@cm01dbm01 ~]$

ASM and Storage Details

As part of our configuration worksheet, the customer provides Oracle with an ASM layout and the installation scripts create ASM disk groups based on information we've provided. Backing up a bit, here's how storage and ASM is defined and allocated with Exadata:

• Each Exadata cell consists of 12 physical disks

• Each physical disk has one LUN created on it, and a small area reserved for the System Area.  The System Area contains the OS image, swap space, Exadata software binaries, metadata repository, and other information.

• Each LUN has a cell disk on it, which is a logical layer of storage available for creating Grid Disks. For the first two disks in each storage cell, Oracle automatically creates a cell disk for all tracks beyond the 29Gb System Area and for the remaining 10 disks, Oracle creates a cell disk for the entire LUN.

• Grid Disks are created on the cell disks and are the layer presented to and available to ASM.

• When Grid Disks are created, but default a like-sized Grid Disk will be created on each LUN on each cell disk.

• ASM disk groups are based on Grid Disks.

• The compute node's ASM instance can see ASM disk and candidate Grid Disks based on the Infiniband interconnect network address specified in cellinit.ora

• ASM disk groups are created on Grid Disks. ASM disk groups can be created on a subset of Grid Disks or the entire set of Grid Disks spanning all physical disks (by way of cell disks and the LUNs on which they reside).

For our Exadata Quarter Rack, we can determine the layout of our ASM disk groups and the physical disks as below:

SQL> conn / as sysdba

Connected.

SQL> select name from v$asm_diskgroup;

NAME

------------------------------

DATA_CM01

DBFS_DG

RECO_CM01

SQL> Disconnected from Oracle Database 11g
Enterprise Edition Release 11.2.0.2.0 - 64bit Production

With the Partitioning, Real Application
Clusters, Automatic Storage Management, OLAP,

Data Mining and Real Application Testing
options

[oracle@cm01dbm01 ~]$ dcli -g cell_group cellcli
-e list griddisk detail > griddisk.out

[oracle@cm01dbm01 ~]$

As we can see from the query against V$ASM_DISKGROUP, we have 3 disk groups: DATA_CM01, RECO_CM01, and DBFS_DG.  DATA_CM01 is for data files, RECO_CM01 is for the Fast Recovery Area (previously known as Flash Recovery Area but re-branded with the advent of Exadata), and DBFS_DG is a disk group created for DBFS. Additionally, as displayed earlier, the OCR and Voting Disks are multiplexed and stored in the DBFS_DG disk group.

When Grid Disks are created in an Exadata storage cell, Oracle carves out space from the outer tracks and works its way in to the inner tracks.  This being the case, the Grid Disks created first will have the best performance characteristics and the disks created last on each cell disk will have the slowest performance. Carrying this to the next level, ASM disk groups created on the outer-most Grid Disks will perform fastest. Looking at the cellcli list griddisk detail output displayed above, we can determine how the storage is physically allocated. Let's look at griddisk.out generated above:

[oracle@cm01dbm01 ~]$ grep "name:"
griddisk.out

cm01cel01: name: DATA_CM01_CD_00_cm01cel01
<= Displays all from cm01cel01 first

cm01cel01: name: DATA_CM01_CD_01_cm01cel01

cm01cel01: name: DATA_CM01_CD_10_cm01cel01

<< rows deleted >>

cm01cel01: name: DATA_CM01_CD_11_cm01cel01
<= Will display a Grid Disk for each of the
12 Cell Disks for each Grid Disk

cm01cel01: name: DBFS_DG_CD_02_cm01cel01
<= Moves on to the next Grid Disk
alphabetically

cm01cel01: name: DBFS_DG_CD_03_cm01cel01

<< rows deleted >>

cm01cel01: name: DBFS_DG_CD_11_cm01cel01

cm01cel01: name: RECO_CM01_CD_00_cm01cel01
<= 3rdGrid Disk, first cell

cm01cel01: name: RECO_CM01_CD_01_cm01cel01

<< rows deleted >>

cm01cel01: name: RECO_CM01_CD_11_cm01cel01

cm01cel02: name: DATA_CM01_CD_00_cm01cel02
<= Begin cm01cel02

<< rows deleted >>

cm01cel03: name: DATA_CM01_CD_00_cm01cel03
<= Begin cm01cel03

<< rows deleted >>

To see how Grid Disks are physically organized on each cell disk:

[oracle@cm01dbm01 ~]$ egrep
"name:|cellDisk|offset" griddisk.out |more

cm01cel01: name: DATA_CM01_CD_00_cm01cel01

cm01cel01: cellDisk: CD_00_cm01cel01

cm01cel01: offset: 32M

cm01cel01: name: DATA_CM01_CD_01_cm01cel01

cm01cel01: cellDisk: CD_01_cm01cel01

cm01cel01: offset: 32M

cm01cel01: name: DATA_CM01_CD_02_cm01cel01

cm01cel01: cellDisk: CD_02_cm01cel01

cm01cel01: offset: 32M

cm01cel01: name: DATA_CM01_CD_03_cm01cel01

cm01cel01: cellDisk: CD_03_cm01cel01

cm01cel01: offset: 32M

cm01cel01: name: DATA_CM01_CD_04_cm01cel01

cm01cel01: cellDisk: CD_04_cm01cel01

cm01cel01: offset: 32M

cm01cel01: name: DATA_CM01_CD_05_cm01cel01

cm01cel01: cellDisk: CD_05_cm01cel01

In this, the offset shows how many bytes offset from the first track that the Grid Disk begins. If we focus on cm01cel01 and the first physical disk, we see this:

[oracle@cm01dbm01 ~]$ egrep
"name:|cellDisk|offset" griddisk.out |grep CD_00|grep cel01

cm01cel01: name: DATA_CM01_CD_00_cm01cel01

cm01cel01: cellDisk: CD_00_cm01cel01

cm01cel01: name: RECO_CM01_CD_00_cm01cel01

cm01cel01: cellDisk: CD_00_cm01cel01

[oracle@cm01dbm01 ~]$

This shows us that we've got a DATA_CM01 and RECO_CM01 Grid Disk created on the first disk, with an offset of 32M and approximately 423Gb respectively. This means that 32Mb into the drive, DATA_CM01 Grid disk begins. The next Grid Disk (RECO_CM01) begins 423Gb into the cell disk, meaning that the size of DATA_CM01 Grid Disk on the first cell disk is (423Gb - 32Mb) in size.

Note that on the first cell disk, there was no Grid Disk created for DBFS_DG.  This is because the System Areas is placed on the innermost 29Gb of storage on the first two disks. Since ASM needs equally sized disks across all Grid Disks, Oracle decides to place the DBFS Grid Disks and their associated ASM disk groups on the inner-most 29Gb of storage on drives 2-11 (excluding drive 0 and 1).

Using some amateurish Perl scripting we can come up with a more readable output from the cellcli command:

#!/usr/bin/perl

open(F,"./griddisk.out");

while (<F>) {

if (/name\:/) {

($x1,$x2,$gdname)=split(":",$_);

chop($gdname);

}

if (/cellDisk\:/) {

($x1,$x2,$celldisk)=split(":",$_);

chop($celldisk);

}

if (/offset\:/) {

($x1,$x2,$offset)=split(":",$_);

chop($offset);

}

if (/size\:/) {

($x1,$x2,$dsize)=split(":",$_);

}

if (/status\:/) {

# push to array

print "$celldisk\t$gdname\t$offset\t$dsize";

}

}

close(F);

InfiniBand Details

The Exadata X2-2 Quarter Rack comes with two Infiniband Switches, in our case named:

cm01sw-ib3.centroid.com

cm01sw-ib2.centroid.com

We can access the switches by navigating to:

http://cm01sw-ib3.centroid.com

http://cm01sw-ib2.centroid.com

These URLs take you to the Infiniband ILOM console. The default username and password are ilom-admin/ilom-admin. I changed the passwords to the same as the other nodes, but for some reason it didn't take.

Our network configuration looks like this:

Infiniband Details

Enterprise Manager Database Control
The installation and deployment process configures Enteprise Manager Database Control:

http://cm01dbm01.centroid.com:1158/em

http://cm01dbm02.centroid.com:1158/em

Enterprise Manager Database Control

ILOM Access

Each compute node and cell server has an ILOM (Integrated Lights Out Management) web interface that can be used to access the node in the event maintenance of troubleshooting is required to manage the server. The URLs are below:

https://cm01dbm01-ilom.centroid.com/iPages/suntab.asp

https://cm01dbm02-ilom.centroid.com/iPages/suntab.asp

https://cm01cel01-ilom.centroid.com/iPages/suntab.asp

https://cm01cel02-ilom.centroid.com/iPages/suntab.asp

https://cm01cel03-ilom.centroid.com/iPages/suntab.asp

The root password is "welcome1" but we changed it on all nodes to the same as the root password on all nodes.

ILOM Access

KVM Access

We can gain console access on each of the compute and cell nodes using the KVM switch, see below:

https://cm01sw-kvm.centroid.com/home.php

Username and password: Admin/<password>

You can login to the console on each of the 5 nodes (but not InfiniBand switch or embedded Cisco switch) using the KVM console.