DBCA does not call 'datapatch' for database changes in 12.1.0.X.

DBCA does not call 'datapatch' for database changes in 12.1.0.X.

After creating a new database make sure to check DB register patch set level.

select PATCH_ID, PATCH_UID, VERSION, STATUS, DESCRIPTION
from DBA_REGISTRY_SQLPATCH
order by BUNDLE_SERIES;

no rows selected

whereas opatch lsinventory shows the patch installed

Patch description: “Database Patch Set Update : 12.1.0.2.5 (21359755)”

Solution

DBCA doesn't execute datapatch in Oracle 12.1.0.X. The solution is to apply the SQL changes manually after creating a new Database .

After creating a new database make sure to run:

cd $ORACLE_HOME/OPatch
./datapatch -verbose

then check DBA_REGISTRY_SQLPATCH view

With a multitenant database datapatch is run for all pluggable databases automatically, which are opened for writing, and for the PDB$SEED as well. No need to run datapatch for PDBs that you create afterwards from the template.

ORA-16857: standby disconnected from redo source for longer than specified threshold

DGMGRL> show configuration

Configuration - usage

  Protection Mode: MaxPerformance
  Members:
  usage   - Primary database
    usagedr - Physical standby database
      Warning: ORA-16857: standby disconnected from redo source for longer than specified threshold

Fast-Start Failover: DISABLED

Configuration Status:
WARNING   (status updated 14 seconds ago)

DGMGRL> show database usage

Database - usage

  Role:               PRIMARY
  Intended State:     TRANSPORT-ON
  Instance(s):
    usage1
    usage2

Database Status:
SUCCESS

DGMGRL> show database usagedr

Database - usagedr

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON
  Transport Lag:      23 minutes 55 seconds (computed 2 seconds ago)
  Apply Lag:          23 minutes 56 seconds (computed 2 seconds ago)
  Average Apply Rate: 1.40 MByte/s
  Real Time Query:    OFF
  Instance(s):
    usagedr1 (apply instance)
    usagedr2

Database Status:
SUCCESS

DGMGRL>

Solution

Setting 'TransportDisconnectedThreshold' to 0 seconds will disable the alert else we can set the property to a higher value.

COMMON CAUSES OF OCSSD EVICTIONS

Oracle Clusterware is designed to perform a node eviction by removing one or more nodes from the cluster if some critical problem is detected.  A critical problem could be a node not responding via a network heartbeat, a node not responding via a disk heartbeat, a hung or severely degraded machine, or a hung ocssd.bin process.  The purpose of this node eviction is to maintain the overall health of the cluster by removing bad members.

Common Causes are as below.

1. Network failure or latency between nodes. It would take 30 consecutive missed checkins (by default - determined by the CSS misscount) to cause a node eviction. 
2. Problems writing to or reading from the CSS voting disk.  If the node cannot perform a disk heartbeat to the majority of its voting files, then the node will be evicted.
3. A member kill escalation.  For example, database LMON process may request CSS to remove an instance from the cluster via the instance eviction mechanism.  If this times out it could escalate to a node kill. 
4. An unexpected failure or hang of the OCSSD process, this can be caused by any of the above issues or something else.
5. An Oracle bug.

At some extents we can maintain them using below 

$CRS_HOME/bin/crsctl set css misscount n -- default 60 sec in 11g nd 30 sec in 12c

$CRS_HOME/bin/crsctl set css reboottime n -- default 3

$CRS_HOME/bin/crsctl set css disktimeout n -- default 200 sec

Set environment variable RAT_DBNAMES to name of the database and re-run

One day i was running exachk, immediately process stoped with below message 

root@escinfra01:/export/home/oracle/exachk# ./exachk

Searching for running databases . . . . .

Databases are not registered in Clusterware to check best practices. Set environment variable RAT_DBNAMES to name of the database and re-run.
eg. like export RAT_DBNAMES="TESTDB,PRODDB"

root@escinfra01:/export/home/oracle/exachk#

Cause

I figured out that this server was holding some databases which are not registered with Cluster as result exachk is terminating and requesting for proper SID's to be set.

Solution

I set RAT_DBNAMES to one of my database which is registered with cluster and re-executed exachk and it through.

root@escinfra01:/export/home/oracle# export RAT_DBNAMES=dbm08
root@escinfra01:/export/home/oracle#
root@escinfra01:/export/home/oracle/exachk# ./exachk
.  .
.  .
Checking Status of Oracle Software Stack - Clusterware, ASM, RDBMS
.
.  .  .  . . . .  .  .  .
-------------------------------------------------------------------------------------------------------
                                                 Oracle Stack Status
-------------------------------------------------------------------------------------------------------
  Host Name       CRS Installed  RDBMS Installed    CRS UP    ASM UP  RDBMS UP    DB Instance Name
-------------------------------------------------------------------------------------------------------
 escinfra01                 Yes          Yes          Yes      Yes       No
-------------------------------------------------------------------------------------------------------

Unable to archive thread 1 sequence ###

From alert log it was noticed that archive process is leaving some messages as below.

Sun Nov 11 09:41:34 2018
ARC5: Evaluating archive log 20 thread 1 sequence 59168
ARC5: Unable to archive thread 1 sequence 59168
      Log actively being archived by another process
ARC5: Evaluating archive log 22 thread 1 sequence 59175
ARC5: Unable to archive thread 1 sequence 59175
      Log actively being archived by another process

Cause

It is due to having more than one archiver.

Solution

If automatic archiving is turned 'ON' and LOG_ARCHIVE_MAX_PROCESSES is set to more than one, then all of the archiver processes will try to archive this logfile, however they will not be able to aquire the lock 'kcrrlt', to protect multiple arch processes from archiving the same logfile.

The failing process process will write to the trace/alert message that it was unable to archive the logfile.

It doesn't mean that the log mentioned is not archived; it is successfully archived by some other process. It only means that the log was not archived the first time it was tried.


If you go a bit down you will see that the log had been archived.

Sun Nov 11 09:41:45 2018
ARC9: Completed archiving thread 1 sequence 59168 (0-0)

ORA-39083: Object type PROCACT_SYSTEM failed to create with error

Error

Processing object type DATABASE_EXPORT/SYSTEM_PROCOBJACT/PRE_SYSTEM_ACTIONS/PROCACT_SYSTEM
>>> Cannot set an SCN larger than the current SCN. If a Streams Capture configuration was imported then the Apply that processes the captured messages needs to be dropp
ed and recreated. See My Oracle Support article number 1380295.1.
ORA-39083: Object type PROCACT_SYSTEM failed to create with error:

ORA-20000: Incompatible version of Workspace Manager Installed

Cause

Version of the Oracle Workspace Manager (OWM) in the source and target are not same.

Solution

Ignore the following errors, if IMPDP is importing all other components and data successfully:

To move the workspaces from the 11.2.0.3 or 11.2.0.4 SOURCE db, the FULL db export must be generated with the parameter "VERSION=12".

If the SOURCE database is older than 11.2.0.3, e.g. 11.2.0.2 or 11.1.0.7, the database will first need to be upgraded to 11.2.0.3 or 11.2.0.4