SQL Net client trace

1. Add following parameters in the sqlnet.ora file in CLIENT machine

TRACE_LEVEL_CLIENT = 16
TRACE_FILE_CLIENT = CLIENT
TRACE_DIRECTORY_CLIENT = D:\TRACE
TRACE_UNIQUE_CLIENT = ON
LOG_FILE_CLIENT = CLIENT
LOG_DIRECTORY_CLIENT = D:\TRACE\LOG
TNSPING.TRACE_DIRECTORY = D:\TRACE\TRACE
TNSPING.TRACE_LEVEL = ADMIN

2. Reproduce the issue to find error

3. Disable tracing by removing above entries from sqlnet.ora after generating trace.

DB_UNIQUE_NAME Conflict on Exadata

I was trying to restore one of our database on Supercluster M7 server and i came across error in controlfile file restore itself.

RMAN> run
{
allocate channel c1 type 'sbt_tape';
SEND 'NB_ORA_SERV=bkppdb01, NB_ORA_CLIENT=monetadb01-bkp';
restore controlfile from 'cntrl_FMREF_bnt98v1v_1_1';
RELEASE CHANNEL c1;
}2> 3> 4> 5> 6> 7>

using target database control file instead of recovery catalog
allocated channel: c1
channel c1: SID=1141 instance=fmref1 device type=SBT_TAPE
channel c1: Veritas NetBackup for Oracle - Release 7.7.3 (2016051915)

sent command to channel: c1

Starting restore at 05-AUG-18

channel c1: restoring control file
released channel: c1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 08/05/2018 14:55:47
ORA-19870: error while restoring backup piece cntrl_FMREF_bnt98v1v_1_1
ORA-19504: failed to create file "+TSTDATA1"
ORA-15045: ASM file name '+TSTDATA1' is not in reference form
ORA-17502: ksfdcre:5 Failed to create file +TSTDATA1

RMAN>

This backup piece was valid so errors surprised me, on investigation i met some informative errors in asm log as below.

Errors in file /u01/app/oratst/diag/rdbms/fmref/fmref1/trace/fmref1_ora_1268.trc:
ORA-15025: could not open disk "o/192.168.XX.XX;192.168.XX.XX/TSTDATA1_CD_06_vsc02celadm02"
Sun Aug 05 14:55:37 2018
WARNING: Write Failed. group:1 disk:18 AU:18379 offset:0 size:16384
path:Unknown disk
         incarnation:0x12 synchronous result:'I/O error'
         subsys:Unknown library krq:0xffffffff7942e520 bufp:0xffffffff7902b000 osderr1:0xf4 osderr2:0x0
         IO elapsed time: 0 usec Time waited on I/O: 0 usec
Sun Aug 05 14:55:37 2018
Errors in file /u01/app/oratst/diag/rdbms/fmref/fmref1/trace/fmref1_ora_1268.trc:
ORA-15025: could not open disk "o/192.168.XX.XX;192.168.XX.XX/TSTDATA1_CD_11_vsc02celadm03"
WARNING: Write Failed. group:1 disk:35 AU:18408 offset:0 size:16384
path:Unknown disk
         incarnation:0x12 synchronous result:'I/O error'
         subsys:Unknown library krq:0xffffffff786b8030 bufp:0xffffffff7902b000 osderr1:0xf4 osderr2:0x0
         IO elapsed time: 0 usec Time waited on I/O: 0 usec
Sun Aug 05 14:55:37 2018
Errors in file /u01/app/oratst/diag/rdbms/fmref/fmref1/trace/fmref1_ora_1268.trc:
ORA-15025: could not open disk "o/192.168.XX.XX;192.168.XX.XX/TSTDATA1_CD_00_vsc02celadm01"
WARNING: Write Failed. group:1 disk:0 AU:18389 offset:0 size:16384
path:Unknown disk
         incarnation:0x12 synchronous result:'I/O error'
         subsys:Unknown library krq:0xffffffff786b7ba8 bufp:0xffffffff7902b000 osderr1:0xf4 osderr2:0x0
         IO elapsed time: 0 usec Time waited on I/O: 0 usec
Sun Aug 05 14:55:37 2018
Errors in file /u01/app/oratst/diag/rdbms/fmref/fmref1/trace/fmref1_ora_1268.trc:
ORA-15080: synchronous I/O operation failed to write block 0 of disk 18 in disk group TSTDATA1
WARNING: failed to write mirror side 1 of virtual extent 0 logical extent 0 of file 332 in group 1 on disk 18 allocation unit 18379
Sun Aug 05 14:55:37 2018
Errors in file /u01/app/oratst/diag/rdbms/fmref/fmref1/trace/fmref1_ora_1268.trc:
ORA-15080: synchronous I/O operation failed to write block 0 of disk 35 in disk group TSTDATA1
WARNING: failed to write mirror side 2 of virtual extent 0 logical extent 1 of file 332 in group 1 on disk 35 allocation unit 18408
Sun Aug 05 14:55:37 2018
Errors in file /u01/app/oratst/diag/rdbms/fmref/fmref1/trace/fmref1_ora_1268.trc:
ORA-15080: synchronous I/O operation failed to write block 0 of disk 0 in disk group TSTDATA1
WARNING: failed to write mirror side 3 of virtual extent 0 logical extent 2 of file 332 in group 1 on disk 0 allocation unit 18389
WARNING: group 1 file 332 vxn 0 block 0 write I/O failed

Towards these errors i have examined and found that the cause of this issue is that, the same name of database was already created in our storage and creating DB with same name again was triggering error from CELL.

Cause: A code change was done in 12.1.2.1.2  that enforced Global DB_UNIQUE_NAME across all virtual hosts sharing cells.

Workaround:

Edit the cellint.ora on each storage cell to add in _cell_db_unique_name_check=false and then restart all cell services. This can be done rolling one cell at a time.

OR

Use different DB_UNIQUE_NAME to avoid modification on CELL level.

MGMTDB manual creation

GI Management Repository configuration tool

oragrid@tabsmed01:~/scripts$ srvctl status mgmtdb
PRCD-1120 : The resource for database _mgmtdb could not be found.
PRCR-1001 : Resource ora.mgmtdb does not exist
oragrid@tabsmed01:~/scripts$

oragrid@tabsmed01:~/scripts$ ./mdbutil.pl --status
mdbutil.pl version : 1.95
2018-08-13 11:31:51: I Checking CHM status...
2018-08-13 11:31:52: W MGMTLSNR is not configured
2018-08-13 11:31:53: W MGMTDB is not configured on tabsmed01!
2018-08-13 11:31:54: I Cluster Health Monitor (CHM) is configured and running
2018-08-13 11:41:54: E Fail to get CHM Repository Path, exiting...
oragrid@tabsmed01:~/scripts$

oragrid@tabsmed01:~/scripts$ ./mdbutil.pl --addmdb --target=+DATA_DG1
mdbutil.pl version : 1.95
2018-08-13 15:14:27: I Starting To Configure MGMTDB at +DATA_DG1...
2018-08-13 15:14:51: I Container database creation in progress...
2018-08-13 15:21:06: I Plugable database creation in progress...
2018-08-13 15:22:38: I Executing "/tmp/mdbutil.pl --addchm" on tabsmed01 as root to configure CHM.
Password:
2018-08-13 15:23:22: I Executing "/tmp/mdbutil.pl --addchm" on tabsmed02 as root to configure CHM.
Password:
2018-08-13 15:23:28: I MGMTDB & CHM configuration done!
oragrid@tabsmed01:~/scripts$

oragrid@tabsmed01:~/scripts$ ./mdbutil.pl --status
mdbutil.pl version : 1.95
2018-08-14 09:29:58: I Checking CHM status...
2018-08-14 09:29:59: I Listener MGMTLSNR is configured and running on tabsmed01
2018-08-14 09:30:01: I Database MGMTDB is configured and running on tabsmed01
2018-08-14 09:30:01: I Cluster Health Monitor (CHM) is configured and running
--------------------------------------------------------------------------------
CHM Repository Path = +DATA_DG1/_MGMTDB/FDB191CA900D2A2CE04400163E5BEA5F/DATAFILE/sysmgmtdata.264.984064885
MGMTDB space used on DG +DATA_DG1 = 4258 Mb
--------------------------------------------------------------------------------
oragrid@tabsmed01:~/scripts$

Summary of ASM Disk Group Attributes

ASM Disk group attributes are parameters that are bound to a disk group, these attributes can be set when a disk group is created or altered.

Here is the list of attributes:

ACCESS_CONTROL.ENABLED
ACCESS_CONTROL.UMASK
AU_SIZE
CELL.SMART_SCAN_CAPABLE
CELL.SPARSE_DG
COMPATIBLE.ASM
COMPATIBLE.RDBMS
COMPATIBLE.ADVM
CONTENT.CHECK
CONTENT.TYPE
DISK_REPAIR_TIME
FAILGROUP_REPAIR_TIME
IDP.BOUNDARY and IDP.TYPE
PHYS_META_REPLICATED
SECTOR_SIZE
STORAGE.TYPE
THIN_PROVISIONED

Lets see an example:

oragrid@irafmdr01:~$ asmcmd lsattr -G DATA_USG -l
Name                        Value
access_control.enabled      FALSE
access_control.umask        066
appliance._partnering_type  EXADATA FIXED
appliance.mode              TRUE
au_size                     4194304
cell.smart_scan_capable     TRUE
cell.sparse_dg              allnonsparse
compatible.asm              12.1.0.2.0
compatible.rdbms            11.2.0.4.0
content.check               FALSE
content.type                data
disk_repair_time            3.6h
failgroup_repair_time       24.0h
idp.boundary                auto
idp.type                    dynamic
logical_sector_size         512
phys_meta_replicated        true
sector_size                 512
thin_provisioned            FALSE
oragrid@irafmdr01:~$

If you want to check from SQL then use the below SQL.

SQL> SELECT dg.name as diskgroup, SUBSTR(a.name,1,20) as name,
     SUBSTR(a.value,1,25) AS value FROM V$ASM_DISKGROUP dg, V$ASM_ATTRIBUTE a
     WHERE dg.group_number = a.group_number
     and a.name like '%smart%';  2    3    4

DISKGROUP                      NAME                     VALUE
------------------------------ ------------------------ ------------------------
DATA_DG1                       cell.smart_scan_capable  TRUE
DBFS_DG1                       cell.smart_scan_capable  TRUE
RECO_DG1                       cell.smart_scan_capable  TRUE

Cluster name

Sometime we might need to check our cluster name in our RAC, lets see how to check it.

login to server using grid user and go to $ORA_CRS_HOME/bin or in some servers we might have set ORA_CRS_HOME as ORACLE_HOME where cluster binaries were installed.

Which ever applicable to your environment switch it.

oragrid@vsc02zdbadm020102:cd $ORA_CRS_HOME/bin

Now simply type ./cemutlo and entry to see the available options.

oragrid@vsc02zdbadm020102:/u01/app/12.1.0.2/grid/bin$ ./cemutlo
Usage: ./cemutlo.bin [-n] [-w]
        where:
        -n prints the cluster name
        -w prints the clusterware version in the following format:
                 <major_version>:<minor_version>:<vendor_info>
oragrid@vsc02zdbadm020102:/u01/app/12.1.0.2/grid/bin$

As we need to know only the cluster name use ./cemutlo -n to see what's our cluster name.

oragrid@vsc02zdbadm020102:/u01/app/12.1.0.2/grid/bin$ ./cemutlo -n
tab-clu5
oragrid@vsc02zdbadm020102:/u01/app/12.1.0.2/grid/bin$

Conclusion:  Here is our cluster name "tab-clu5" by default cluster name is "crs".