Output file was found but is zero sized - Deleted

Output file was found but is zero sized - Deleted

I go a call from our developer that request are completing in warning status.

On investigation i found blow errors in OPP logs.

Error

[092618_131735369][][EXCEPTION] [DEBUG]  [sun.cpu.isalist]:[]
[9/26/18 1:17:35 PM] [1648558:RT19832034] Output file was found but is zero sized - Deleted
[9/26/18 1:17:35 PM] [UNEXPECTED] [1648558:RT19832034] java.io.FileNotFoundException: /d01/Prod/temp_xml/xdo6QGDTbJCca092618_1317356982.fo (Permission denied)
        at java.io.FileOutputStream.open(Native Method)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
        at oracle.apps.xdo.common.tmp.TmpFile.createTmpFileJDK118(TmpFile.java:146)
        at oracle.apps.xdo.common.tmp.TmpFile.createTmpFile(TmpFile.java:113)
        at oracle.apps.xdo.template.fo.util.FOUtility.generateFO(FOUtility.java:987)
        at oracle.apps.xdo.template.fo.util.FOUtility.generateFO(FOUtility.java:212)
        at oracle.apps.xdo.template.FOProcessor.createFO(FOProcessor.java:1647)
        at oracle.apps.xdo.template.FOProcessor.generate(FOProcessor.java:941)
        at oracle.apps.xdo.oa.schema.server.TemplateHelper.runProcessTemplate(TemplateHelper.java:5936)
        at oracle.apps.xdo.oa.schema.server.TemplateHelper.processTemplate(TemplateHelper.java:3459)
        at oracle.apps.xdo.oa.schema.server.TemplateHelper.processTemplate(TemplateHelper.java:3548)
        at oracle.apps.fnd.cp.opp.XMLPublisherProcessor.process(XMLPublisherProcessor.java:302)
        at oracle.apps.fnd.cp.opp.OPPRequestThread.run(OPPRequestThread.java:176)

[9/26/18 1:17:35 PM] [1648558:RT19832034] Completed post-processing actions for request 19832034.


Cause 

The Temporary directory for XML Publisher has not been modified after making the clone so it was pointing to non existing location and getting error out.

Solution

1. Create a new Temporary Directory on OS level for the cloned environment.
2. Make sure that the application owner (APPLMGR) has Read and Write permission this directory.
3. Setup this new directory for XML Publisher via : XML Publisher Administrator responsibility: Properties -> General -> Temporary Directory.
3. Restart the Server.
4. Test again with a XML Publisher Report.

Disable BI Publisher Enterprise Manager 13c

Enterprise Manager 13c – Disable the BI Publisher

The BI Publisher will be started automatically during the startup process of the Enterprise Manager 13c. You don’t like the BI Publisher or you don’t use it? Save the resources, speed up your startup process, disable it. The password of the database repository owner SYSMAN is required.

Verify the Status – the BI Publisher is up and running

[oracle@vivaoem bin]$ ./emctl status oms
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up
BI Publisher Server is Up
[oracle@vivaoem bin]$

Yes it was up, lets disable it.

[oracle@vivaoem bin]$ ./emctl config oms -disable_bip
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
BI Publisher Server is Down
BI Publisher has been disabled on this host and will not be started with the 'emctl start oms' or 'emctl start oms -bip_only' commands.
Overall result of operations: SUCCESS
[oracle@vivaoem bin]$
[oracle@vivaoem bin]$ ./emctl status oms
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up
BI Publisher Server is Down
BI Publisher is disabled, to enable BI Publisher on this host, use the 'emctl config oms -enable_bip' command
[oracle@vivaoem bin]$

Please note its just an temporary disable, on future restart of EM services BIP will be started again, you need to disable it again.

RMAN-06026: some targets not found - aborting restore

I was cloning one of our TEST and server and stuckup on "RMAN-06026 and RMAN-06023" while restoring database.

RMAN> run
{
SET NEWNAME FOR DATABASE   TO  '/u01/oracle/app/gbldrsdata/%b';
SET NEWNAME FOR tempfile  1 TO '/u01/oracle/app/gbldrsdata/%b';
restore database ;
switch datafile all;
switch tempfile all;
set until sequence 20686 thread 1;
recover database;
}2> 3> 4> 5> 6> 7> 8> 9> 10>

executing command: SET NEWNAME
executing command: SET NEWNAME

Starting restore at 10-AUG-18
using channel ORA_DISK_1
using channel ORA_DISK_2

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 08/10/2018 00:52:27
RMAN-06026: some targets not found - aborting restore
RMAN-06023: no backup or copy of datafile 4 found to restore
RMAN-06023: no backup or copy of datafile 3 found to restore
RMAN-06023: no backup or copy of datafile 2 found to restore
RMAN-06023: no backup or copy of datafile 1 found to restore

RMAN>

After drill-down i found that there are some backup files in FRA that belongs to OLD INCARNATION, as i was not interested in these files i have disabled FRA.

commented *.db_recovery_file_dest and *.db_recovery_file_dest_size in pfile and restarted restore and it went successful.

Function not available to this responsibility When Trying to Access Custom Form

When attempting to access custom forms, the following error occurs.

"Function not available to this responsibility. Change Responsibilities or contact your System Administrator".

Cause: Missing entry of CUSTOM_TOP under default.env file.

Solution:

Below procedure requrired to function cutom entry

1. Add  missing CUSTOM_TOP entry to $INST_TOP/ora/10.1.2/forms/server/default.env
XX_TOP=/d01/Prod/apps/apps_st/appl/xx_top/12.0.0

2. Add XX_TOP entry to CONTEXT_FILE

<XX_TOP oa_var="s_xxtop" oa_type="PROD_TOP" oa_enabled="FALSE">/d01/Prod/apps/apps_st/appl/xx_top/12.0.0</XX_TOP>

3. source APPSPROD_XXX.env

4. Restart Middle Tier services.

SQL Net client trace

1. Add following parameters in the sqlnet.ora file in CLIENT machine

TRACE_LEVEL_CLIENT = 16
TRACE_FILE_CLIENT = CLIENT
TRACE_DIRECTORY_CLIENT = D:\TRACE
TRACE_UNIQUE_CLIENT = ON
LOG_FILE_CLIENT = CLIENT
LOG_DIRECTORY_CLIENT = D:\TRACE\LOG
TNSPING.TRACE_DIRECTORY = D:\TRACE\TRACE
TNSPING.TRACE_LEVEL = ADMIN

2. Reproduce the issue to find error

3. Disable tracing by removing above entries from sqlnet.ora after generating trace.

DB_UNIQUE_NAME Conflict on Exadata

I was trying to restore one of our database on Supercluster M7 server and i came across error in controlfile file restore itself.

RMAN> run
{
allocate channel c1 type 'sbt_tape';
SEND 'NB_ORA_SERV=bkppdb01, NB_ORA_CLIENT=monetadb01-bkp';
restore controlfile from 'cntrl_FMREF_bnt98v1v_1_1';
RELEASE CHANNEL c1;
}2> 3> 4> 5> 6> 7>

using target database control file instead of recovery catalog
allocated channel: c1
channel c1: SID=1141 instance=fmref1 device type=SBT_TAPE
channel c1: Veritas NetBackup for Oracle - Release 7.7.3 (2016051915)

sent command to channel: c1

Starting restore at 05-AUG-18

channel c1: restoring control file
released channel: c1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 08/05/2018 14:55:47
ORA-19870: error while restoring backup piece cntrl_FMREF_bnt98v1v_1_1
ORA-19504: failed to create file "+TSTDATA1"
ORA-15045: ASM file name '+TSTDATA1' is not in reference form
ORA-17502: ksfdcre:5 Failed to create file +TSTDATA1

RMAN>

This backup piece was valid so errors surprised me, on investigation i met some informative errors in asm log as below.

Errors in file /u01/app/oratst/diag/rdbms/fmref/fmref1/trace/fmref1_ora_1268.trc:
ORA-15025: could not open disk "o/192.168.XX.XX;192.168.XX.XX/TSTDATA1_CD_06_vsc02celadm02"
Sun Aug 05 14:55:37 2018
WARNING: Write Failed. group:1 disk:18 AU:18379 offset:0 size:16384
path:Unknown disk
         incarnation:0x12 synchronous result:'I/O error'
         subsys:Unknown library krq:0xffffffff7942e520 bufp:0xffffffff7902b000 osderr1:0xf4 osderr2:0x0
         IO elapsed time: 0 usec Time waited on I/O: 0 usec
Sun Aug 05 14:55:37 2018
Errors in file /u01/app/oratst/diag/rdbms/fmref/fmref1/trace/fmref1_ora_1268.trc:
ORA-15025: could not open disk "o/192.168.XX.XX;192.168.XX.XX/TSTDATA1_CD_11_vsc02celadm03"
WARNING: Write Failed. group:1 disk:35 AU:18408 offset:0 size:16384
path:Unknown disk
         incarnation:0x12 synchronous result:'I/O error'
         subsys:Unknown library krq:0xffffffff786b8030 bufp:0xffffffff7902b000 osderr1:0xf4 osderr2:0x0
         IO elapsed time: 0 usec Time waited on I/O: 0 usec
Sun Aug 05 14:55:37 2018
Errors in file /u01/app/oratst/diag/rdbms/fmref/fmref1/trace/fmref1_ora_1268.trc:
ORA-15025: could not open disk "o/192.168.XX.XX;192.168.XX.XX/TSTDATA1_CD_00_vsc02celadm01"
WARNING: Write Failed. group:1 disk:0 AU:18389 offset:0 size:16384
path:Unknown disk
         incarnation:0x12 synchronous result:'I/O error'
         subsys:Unknown library krq:0xffffffff786b7ba8 bufp:0xffffffff7902b000 osderr1:0xf4 osderr2:0x0
         IO elapsed time: 0 usec Time waited on I/O: 0 usec
Sun Aug 05 14:55:37 2018
Errors in file /u01/app/oratst/diag/rdbms/fmref/fmref1/trace/fmref1_ora_1268.trc:
ORA-15080: synchronous I/O operation failed to write block 0 of disk 18 in disk group TSTDATA1
WARNING: failed to write mirror side 1 of virtual extent 0 logical extent 0 of file 332 in group 1 on disk 18 allocation unit 18379
Sun Aug 05 14:55:37 2018
Errors in file /u01/app/oratst/diag/rdbms/fmref/fmref1/trace/fmref1_ora_1268.trc:
ORA-15080: synchronous I/O operation failed to write block 0 of disk 35 in disk group TSTDATA1
WARNING: failed to write mirror side 2 of virtual extent 0 logical extent 1 of file 332 in group 1 on disk 35 allocation unit 18408
Sun Aug 05 14:55:37 2018
Errors in file /u01/app/oratst/diag/rdbms/fmref/fmref1/trace/fmref1_ora_1268.trc:
ORA-15080: synchronous I/O operation failed to write block 0 of disk 0 in disk group TSTDATA1
WARNING: failed to write mirror side 3 of virtual extent 0 logical extent 2 of file 332 in group 1 on disk 0 allocation unit 18389
WARNING: group 1 file 332 vxn 0 block 0 write I/O failed

Towards these errors i have examined and found that the cause of this issue is that, the same name of database was already created in our storage and creating DB with same name again was triggering error from CELL.

Cause: A code change was done in 12.1.2.1.2  that enforced Global DB_UNIQUE_NAME across all virtual hosts sharing cells.

Workaround:

Edit the cellint.ora on each storage cell to add in _cell_db_unique_name_check=false and then restart all cell services. This can be done rolling one cell at a time.

OR

Use different DB_UNIQUE_NAME to avoid modification on CELL level.

MGMTDB manual creation

GI Management Repository configuration tool

oragrid@tabsmed01:~/scripts$ srvctl status mgmtdb
PRCD-1120 : The resource for database _mgmtdb could not be found.
PRCR-1001 : Resource ora.mgmtdb does not exist
oragrid@tabsmed01:~/scripts$

oragrid@tabsmed01:~/scripts$ ./mdbutil.pl --status
mdbutil.pl version : 1.95
2018-08-13 11:31:51: I Checking CHM status...
2018-08-13 11:31:52: W MGMTLSNR is not configured
2018-08-13 11:31:53: W MGMTDB is not configured on tabsmed01!
2018-08-13 11:31:54: I Cluster Health Monitor (CHM) is configured and running
2018-08-13 11:41:54: E Fail to get CHM Repository Path, exiting...
oragrid@tabsmed01:~/scripts$

oragrid@tabsmed01:~/scripts$ ./mdbutil.pl --addmdb --target=+DATA_DG1
mdbutil.pl version : 1.95
2018-08-13 15:14:27: I Starting To Configure MGMTDB at +DATA_DG1...
2018-08-13 15:14:51: I Container database creation in progress...
2018-08-13 15:21:06: I Plugable database creation in progress...
2018-08-13 15:22:38: I Executing "/tmp/mdbutil.pl --addchm" on tabsmed01 as root to configure CHM.
Password:
2018-08-13 15:23:22: I Executing "/tmp/mdbutil.pl --addchm" on tabsmed02 as root to configure CHM.
Password:
2018-08-13 15:23:28: I MGMTDB & CHM configuration done!
oragrid@tabsmed01:~/scripts$

oragrid@tabsmed01:~/scripts$ ./mdbutil.pl --status
mdbutil.pl version : 1.95
2018-08-14 09:29:58: I Checking CHM status...
2018-08-14 09:29:59: I Listener MGMTLSNR is configured and running on tabsmed01
2018-08-14 09:30:01: I Database MGMTDB is configured and running on tabsmed01
2018-08-14 09:30:01: I Cluster Health Monitor (CHM) is configured and running
--------------------------------------------------------------------------------
CHM Repository Path = +DATA_DG1/_MGMTDB/FDB191CA900D2A2CE04400163E5BEA5F/DATAFILE/sysmgmtdata.264.984064885
MGMTDB space used on DG +DATA_DG1 = 4258 Mb
--------------------------------------------------------------------------------
oragrid@tabsmed01:~/scripts$