Wednesday, October 10, 2007

Troubleshooting after Upgrade Oracle Applications 11i to 10g

Following the last post on Troubleshooting during Upgrade Oracle Applications 11i to 10g, in this post I'm going to write about troubleshooting AFTER upgrade Oracle Applications 11i to 10g.

Case 1: ORA-29877: FAILED IN THE EXECUTION OF THE ODCIINDEXPDATE ROUTINE

Symptom:

When trying to save in "Find Service Request" Form, an error message shows up:

ORA-29877: failed in the execution of the ODCIINDEXUPDATE routine
ORA-20000: Oracle Text error:
DBG-50857: oracle error in textindexmethods. ODCIIndexUpdate
ORA-20000: Oracle Text error:
DBG-10602: failed to queue DML change to column for in Package
CS_ServiceRequest_PVT Procedure Update_ServiceRequest

Cause:

KOREAN_LEXER was deprecated in release 9.0.1 and desupported in release 10.2.0.1.
It is replaced by KOREAN_MORPH_LEXER and Oracle Text indexes need to be migrated from the obsolete KOREAN_LEXER to the new KOREAN_MORPH_LEXER.
This was not done during the database upgrade.

Solution:

Run the following script as sysdba:
sqlplus "/ as sysdba" @$ORACLE_HOME/ctx/sample/script/drkorean.sql

If you get this error (or similar):

$ sqlplus "/as sysdba" @drkorean.sql
SQL*Plus: Release 10.2.0.3.0 - Production on …
Copyright (c) 1982, 2006, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Production
With the Partitioning, OLAP and Data Mining options

List of indexes that use KOREAN_LEXER as top level lexer:
List of indexes that use KOREAN_LEXER as a sub lexer:
"CS"."CS_FORUM_MESSAGES_TL_N4"
"CS"."CS_INCIDENTS_ALL_TL_N1"
"CS"."CS_KB_SOLN_CAT_TL_N1"
"CS"."SUMMARY_CTX_INDEX"
"ICX"."ICX_QUES_CTX"
Migrate KOREAN_LEXER to KOREAN_MORPH_LEXER
Rebuild all indexes that use korean lexer as top level lexer:
Reindex all documents that use KOREAN_LEXER as sub lexer
reindexing : "CS"."CS_FORUM_MESSAGES_TL" finished.
reindexing : "CS"."CS_INCIDENTS_ALL_TL" finished.
declare
*
ERROR at line 1:
ORA-29861: domain index is marked LOADING/FAILED/UNUSABLE
ORA-06512: at line 316

It is mean that one or more Text indexes are invalid so we have to check which indexes are affected.

Run this query to check which indexes are invalid:

select owner, index_name, index_type, status, domidx_status, domidx_opstatus
from dba_indexes
where ityp_owner='CTXSYS'
order by 1;

select idx_owner, idx_name, idx_type, idx_status
from ctxsys.ctx_indexes
order by 1;


Drop the affected indexes (e.g. drop index cs.CS_KB_SOLN_CAT_TL_N1 force;)

Recreate dropped indexes as described in Metalink Note 312640.1 - Oracle Text: Re-installation of Applications 11i (11.5.10) Oracle Text Indexes
Section 2: Create the CS Text index preferences and indexes (7, 8, 9, 10 & 11)

Rerun script $ORACLE_HOME/ctx/sample/script/drkorean.sql to complete the lexer conversion.


Case 2: FRM-40735: ON-ERROR TRIGGER ORA-01001 FRM-42100 WHEN SEARCHING PARTIALLY PEOPLE

Symptom:

In Users form, when trying to query by person field like ‘%name%’ an error message shows up:

FRM-40735: ON-ERROR TRIGGER RAISED UNHANDLED EXCEPTION ORA-01001

At alert_SID.log:
ORA-07445: exception encountered: core dump [qerixGetKey()+562] [SIGSEGV] [Address not mapped to object] [0x10] [] []

Cause:

Initialization parameters didn’t set properly for 10g.

Solution:

Add the follwing to initSID.ora file:
_b_tree_bitmap_plans = FALSE

For additional information see note 216205.1 - Database Initialization Parameters for Oracle Applications 11i


Case 3: Error in iAS Cache: java.io.NotSerializableException: oracle.apps.qot.util.LookupValuesKey

Symptom:

When trying to access the Quote tab from sales online, an error message is returned:

Error in iAS Cache: java.io.NotSerializableException: oracle.apps.qot.util.LookupValuesKey

Restart Apache resolve the issue for a short time but after some clicks it’s happens again.

Solution:

Should apply patch 6024690.
Note 422499.1 - LookupValuesKey Error Accessing Quote Tab


Case 4: Oracle Forms Web CGI: Error detected

Symptom:

After login to applications and choose a responsibility, just before java applet startup, an error message shows up:

Oracle Forms Web CGI: Error detected
Your request cannot be serviced at this time due to the following error:
ERROR: URL Contains One or More Restricted Characters

Solution:

1) Comment out the value FORMS60_BLOCK_URL_CHARACTERS in apps.conf.
2) Bounce Apache.

Follow Note:395628.1 - 11.5.10 Error: URL Contains One Or More Restricted Characters

You are welcom to leave a comment...

Aviad

Wednesday, October 3, 2007

Troubleshooting during Upgrade Oracle Applications 11i to 10g

During the Oracle Apps Database upgrade to 10g that I wrote about in the previous post, I ran into some errors.
In this post I will write about the errors symptoms, causes and the solutions that worked for me. :-)

Case 1: XDB (Oracle XML Database) schema is invalid

Symptom:

During the upgrade process an error message shows up:
ORA-06576: not a valid function or procedure name

In Oracle_Server.log: ($ORACLE_HOME/cfgtoollogs/dbua/[SID]/upgrade/)
call dbms_xdb.CleanSGAForUpgrade();
call dbms_xdb.CleanSGAForUpgrade();
*
ERROR at line 1:
ORA-06576: not a valid function or procedure name

When click on “Ignore” an error message shows up:


ORA-31159: XML DB is in an invalid state
ORA-00600: internal error code, arguments:
[qmtb_init_len], [], [], [], [], [],

Cause:

Oracle XML Database (XDB) is in an Invalid state.

Execute this query to check the XDB state:
select comp_name, status, version
from DBA_REGISTRY
where comp_name= 'Oracle XML Database’;

COMP_NAME STATUS VERSION
Oracle XML Database INVALID 9.2.0.6.0

Solution:

The XDB schema should be reinstalled.
I’ve Followed Note 243554.1 - How to Deinstall and Reinstall XML Database (XDB)

Step by Step Action Plan:
1) Restart the DB
2) Connect to DB as sysdba
3) Run $ORACLE_HOME/rdbms/admin/catnoqm.sql to remove XDB from database.
4) Update init.ora with the following (at least):
Shared_pool_size=150MB
Java_pool_size=150MB
5) Shutdown immediate
6) Startup
7) Run $ORACLE_HOME/rdbms/admin/catqm.sql XDB XDB TEMP
to create the XDB schema with all required objects
8) Run $ORACLE_HOME/rdbms/admin/catxdbj.sql
to register XDB java api jar files.
9) Check invalid objects for xdb users
select count(*)
from dba_objects
where owner='XDB'
and status='INVALID';
10) Check dba_registry (should be valid now…)
select comp_name, status, version
from DBA_REGISTRY
where comp_name= 'Oracle XML Database’;


Case 2: ORA-06512 AT SYS.UTL_RECOMP

Symptoms:

During the upgrade process, an error message shows up:

ORA-01003: no statement parsed
ORA-06512: at "SYS.UTL_RECOMP", line 464
ORA-06512: at "SYS.UTL_RECOMP", line 467
ORA-06512: at "SYS.UTL_RECOMP", line 607
ORA-01003: no statement parsed
ORA-06512: at line 4


I’ve take a look on log file $ORACLE_HOME/cfgtoollogs/dbua/[SID]/upgrade/PostUpgrade.log:

DECLARE
2 threads pls_integer := &&1;
3 BEGIN
4 utl_recomp.recomp_parallel(threads);
5 END;
6 /
DECLARE
*
ERROR at line 1:
ORA-01003: no statement parsed
ORA-06512: at "SYS.UTL_RECOMP", line 464
ORA-06512: at "SYS.UTL_RECOMP", line 467
ORA-06512: at "SYS.UTL_RECOMP", line 607
ORA-01003: no statement parsed
ORA-06512: at line 4

And those are the last lines of the related trace file:

[Thread-23] [18:12:10:580] [SQLEngine.setSpool:1744] Setting Spool =
/oracle/tst10db/10.2/cfgtoollogs/dbua/TST10/upgrade/PostUpgrade.log
[Thread-23] [18:12:10:581] [SQLEngine.setSpool:1745] Is spool appendable? -->
true
[Thread-23] [18:12:10:584] [SQLEngine.reInitialize:602] Reinitializing SQLEngine...
[Thread-23] [18:12:10:591] [SQLEngine.initialize:241] Execing SQLPLUS/SVRMGR process...
[Thread-23] [18:12:10:609] [SQLEngine.initialize:269] m_bReaderStarted: false
[Thread-23] [18:12:10:610] [SQLEngine.initialize:273] Starting Reader Thread...
[Thread-23] [18:12:10:617] [SQLEngine.initialize:322] Waiting for m_bReaderStarted to be true
[Thread-57] [18:58:44:357] [BasicStep.handleNonIgnorableError:430]
oracle.sysman.assistants.util.UIMessageHandler@c8e4de:messageHandler
[Thread-57] [18:58:44:361] [BasicStep.handleNonIgnorableError:431] ORA-01003: no
statement parsed
ORA-06512: at "SYS.UTL_RECOMP", line 464
ORA-06512: at "SYS.UTL_RECOMP", line 467
ORA-06512: at "SYS.UTL_RECOMP", line 607
ORA-01003: no statement parsed
ORA-06512: at line 4

Cause:

Database initialization parameters are not set properly.

Solution:

Set the following initialization parameters:

SHARED_POOL_SIZE = 250 Mb or greater
LARGE_POOL_SIZE = 150 Mb or greater
JAVA_POOL_SIZE = 150 Mb or greater

Then rerunning:

sqlplus "/ as sysdba"
SQL> shutdown immediate
SQL> startup upgrade
SQL> spool my_upgrade.log
SQL> @/rdbms/admin/catpupgrd.sql
SQL> @/rdbms/admin/utlrp.sql
SQL> spool off


Case 3: Shutdown Database Error ORA-600 [Librarycachenotemptyonclose]

Symptom:

The first shutdown after upgrade to 10g shows an error message:

SQL> shutdown immediate
Database closed.
Database dismounted.
ORA-00600: internal error code, arguments: [LibraryCacheNotEmptyOnClose], [], [], [], [], [], [], []

Cause:

Related note 435926.1 - Shutdown Database Erroring ORA-600 [Librarycachenotemptyonclose]

An unpublished bug: 4483084 say that this particular ORA-600 error can be ignored as it means nothing - actually there is no error.

Solution:

Do nothing…
When it will be available, upgrade to 10.2.0.4.0.


Case 4: OLAP Catalog component (AMD) is invalid after upgrade

Symptom:

Check OLAP catalog state:
select comp_name, status, version
from DBA_REGISTRY
where comp_name= 'OLAP Catalog';


COMP_NAME STATUS VERSION
OLAP Catalog INVALID 10.2.0.3.0

Solution:

Recreate OLAP Catalog.

ORACLE_HOME=/oracle/livedb/10.2
ORACLE_SID=TST10
PATH=$ORACLE_HOME/bin:$PATH
sqlplus /nolog
SQL>spool olap_spool.log
SQL>connect SYS as SYSDBA
SQL>@?/olap/admin/olap.sql SYSAUX TEMP;
-- Recompile everything
SQL>@?/rdbms/admin/utlrp.sql
SQL>spool off

Check status of AMD again – should be VALID:
select comp_name,version,status
from dba_registry
where comp_name like '%OLAP%';



Case 5: A lot of invalid objects in DB after upgrade

Symptom:

A lot of invalid objects in DB which cannot be compiled because of this error: (examples)

Compilation errors for PACKAGE BODY APPS.HR_DM_LIBRARY
#13#10Error: PLS-00907: cannot load library unit PUBLIC.DBMS_SQL (referenced by APPS.HR_DM_LIBRARY)
Line: 0
Compilation errors for PACKAGE BODY APPS.HR_DU_DI_INSERT
#13#10Error: PLS-00907: cannot load library unit PUBLIC.UTL_FILE (referenced by APPS.HR_DU_DI_INSERT)
Line: 0
Compilation errors for PACKAGE BODY APPS.IBE_QUOTE_SAVE_PVT
#13#10Error: PLS-00907: cannot load library unit PUBLIC.UTL_HTTP (referenced by
APPS.CZ_CF_API)
Line: 0

Cause:

Existing PL/SQL modules should recompiled in the FORMAT required by the NEW DATABASE.
The utlirp.sql hasn't been run after the upgrade.

Solution:

conn /as sysdba
shutdown
startup upgrade
@$ORACLE_HOME/rdbms/admin/utlirp.sql
shutdown
startup
@$ORACLE_HOME/rdbms/admin/utlrp.sql

About the utlirp.sql script:
The utlirp.sql script invalidate all existing PL/SQL modules (procedures, functions, packages, types, triggers, views) in a database.
This script must be run when it is necessary to regenerate the
compiled code because the PL/SQL code format is inconsistent with the Oracle executable.
Please note that this script does not recompile invalid objects automatically so you must restart the database and explicitly invoke utlrp.sql to recompile invalid objects.

Hope you won't get into this errors, but if so, you can use this post as a reference... :-)

Feel free to leave a comment or to share your experience with 10g upgrade!

You can also register for email subscription to this blog using the box at the right side menu bar.

Aviad

Tuesday, September 25, 2007

Login Page return "Page Not Found" Error

Today I cloned a new test environment on a new server, after all services startup I tried to get the login page by http://ServerName.domain:port/OA_HTML/AppsLocalLogin.jsp but I got the error message “The page cannot be found”.
I tried also http://ServerName.domain:port and I got the same error message.

The Apache error_log show: (55.66.77.88 – server ip, 77.88.1.99 – my ip)

[Mon Sep 24 10:57:13 2007] [error] OPM: EW: Fail to start process with mod=JServ and grp=DiscoGroup, it's possible that your configuration file is not correct.
[Mon Sep 24 10:57:13 2007] [error] OPM: EW: Fail to start process with mod=JServ and grp=OACoreGroup, it's possible that your configuration file is not correct.
[Mon Sep 24 10:57:13 2007] [error] OPM: EW: Fail to start process with mod=JServ and grp=XmlSvcsGrp, it's possible that your configuration file is not correct.


[Mon Sep 24 12:07:00 2007] [error] [client 55.66.77.88] client denied by server configuration: /oracle/testcomn/portal/TEST_servername/oprocmgr-service
[Mon Sep 24 12:07:01 2007] [error] [client 55.66.77.88] client denied by server configuration: /oracle/testcomn/portal/TEST_servername/oprocmgr-service
[Mon Sep 24 12:07:01 2007] [error] [client 55.66.77.88] client denied by server configuration: /oracle/testcomn/portal/TEST_servername/oprocmgr-service
[Mon Sep 24 12:07:01 2007] [error] [client 55.66.77.88] client denied by server configuration: /oracle/testcomn/portal/TEST_servername/oprocmgr-service


[Mon Sep 24 12:14:03 2007] [error] OPM:Can not find one alive process
[Mon Sep 24 12:14:03 2007] [error] [client 77.88.1.99] File does not exist:


Client denied by server configuration??!! Why?!

I checked all Apache configuration files - httpd.conf, httpds.conf, apps.conf, oprocmgr.conf, httpd_pls.conf - but they all were good…
Because it’s a new server I also checked required Linux RPM’s that may be missing, but they all were installed.

The content of /etc/hosts file was:
127.0.0.1 localhost.localdomain localhost ServerName
55.66.77.88 ServerName



We change it to:
127.0.0.1 localhost.localdomain localhost
55.66.77.88 ServerName.domain ServerName


After restart Apache (+ clear cache $COMMON_TOP/_pages/_oa__html/) we got the login screen! :-)

Unfortunately, just after resolving this issue I found a related note in Metalink:
Note:372096.1 - Servlet and JSP Pages return a "Page Not Found" Error After Install of Applications.

Aviad