Corruption in redo log file when implementing Physical Standby

Lately I started implementing Data Guard - Physical Standby - as a DRP environment for our production E-Businsess Suite database and I must share with you one issue I encountered during implementation. I chose one of our test environments as a primary instance and I used a new server, which was prepared to the standby database in production, as the server for the standby database in test. Both are Red-Hat enterprise linux 4. The implementation process went fast with no special issues (at lease I thought so...), everything seems to work fine, archived logs were transmitted from the primary server to the standby server and successfully applied on the standby database. I even executed switchover to the standby server (both database and application tier), and switchover back to the primary server with no problems. The standby database was configured for maximum performance mode, I also created standby redo log files and LGWR was set to asynchronous (ASYNC) network transmission. The exact setting from init.ora file: log_archive_dest_2='SERVICE=[SERVICE_NAME] LGWR ASYNC=20480 OPTIONAL REOPEN=15 MAX_FAILURE=10 NET_TIMEOUT=30' At this stage, when the major part of the implementation had been done, I found some time to deal with some other issues, like interfaces to other systems, scripts, configure rsync for concurrent log files, etc... , and some modifications to the setup document I wrote during implementation. While doing those other issues, I left the physical standby instance active so archive log files are transmitted and applied on the standby instance. After a couple of hours I noticed the following error in the primary database alert log file: ARC3: Log corruption near block 146465 change 8181238407160 time ? Mon Mar  2 13:04:43 2009 Errors in file [ORACLE_HOME]/admin/[CONTEXT_NAME]/bdump/[sid]_arc3_16575.trc: ORA-00354: corrupt redo log block header ORA-00353: log corruption near block 146465 change 8181238407160 time 02/03/2009 11:57:54 ORA-00312: online log 3 thread 1: '[logfile_dir]/redolog3.ora' ARC3: All Archive destinations made inactive due to error 354 Mon Mar  2 13:04:44 2009 ARC3: Closing local archive destination LOG_ARCHIVE_DEST_1: '[archivelog_dir]/arch_[xxxxx].arc' (error 354)([SID]) Committing creation of archivelog '[archivelog_dir]/arch_[xxxxx].arc' (error 354) ARCH: Archival stopped, error occurred. Will continue retrying Mon Mar  2 13:04:45 2009 ORACLE Instance [SID] - Archival Error I don't remember if I've ever had a corruption in redo log file before...  What is wrong?! Is it something with the physical standby instance ?? Actually, if it's something with the standby instance I would have expected for a corruption in the standby redo log files not the primary's.. The primary instance resides on a Netapp volume, so I checked the mount option in /etc/fstab but they were fine. I asked our infrastructure team to check if something went wrong with the network during the time I got the corruption, but they reported that there was no error or something unusual. Ok, I had no choice but to reconstruct the physical standby database, since when an archive log file is missing, the standby database is out of sync'. I set the 'log_archive_dest_state_2' to defer so no further archive log will be transferred to the standby server, cleared the corrupted redo log files (alter database clear unarchived logfile 'logfile.log') and reconstruct the physical standby database. Meanwhile (copy database files takes long...), I checked documentation again, maybe I missed something, maybe I configured something wrong.. I have read a lot and didn't find anything that can shed some light on this issue. At this stage, the standby was up and ready. First, I held up the redo transport service (log_archive_dest_state_2='defer') to see if I'll get a corruption when standby is off.  After one or two days with no corruption I activated the standby. Then I saw the following sentence in Oracle® Data Guard Concepts and Administration 10g Release 2 (10.2): "All members of a Data Guard configuration must run an Oracle image that is built for the same platform. For example, this means a Data Guard configuration with a primary database on a 32-bit Linux on Intel system can have a standby database that is configured on a 32-bit Linux on Intel system" One moment, I thought to myself, the standby server is based on AMD processors and the primary server is based on Intel's..    Is it the problem?! When talking about same platform, is the meaning same processors also? Isn't it sufficient to have same 32 bit OS on x86 machines? Weird but I had to check it... Meanwhile, I got a corruption in redo log file again which assured there is a real problem and it wasn't accidentally. So I used another AMD based server (identical to the standby server) and started all over again – primary and standby instances. After two or three days with no corruption I started to believe the difference in the processors was the problem. But one day later I got a corruption again (Oh no…) I must say that on the one hand I was very frustrated, but on the other hand it was a relief to know it's not the difference in the processors. It was so clear that when I'll find out the problem it will be something stupid.. So it is not the processors, not the OS and not the network.  What else can it be?! And here my familiarity with the "filesystemio_option" initialization parameter begins (thanks to Oracle Support!). I don't know how I missed this note before, but all is written here - Note 437005.1: Redo Log Corruption While Using Netapps Filesystem With Default Setting of Filesystemio_options Parameter. When the redo log files are on a netapp volume, "filesystemio_options" must be set to "directio" (or "setall"). When "filesystemio_options" is set to "none" (like my instance before), read/writes to the redo log files are using the OS buffer cache. Since netapp storage is based on NFS (which is stateless protocol), when performing asynchronous writing over the network, the consistency of writes is not guaranteed. Some writes can be lost. By setting the "filesystemio_options" to "directio", writes bypasses the OS cache layer so no write will be lost. Needless to say that when I set it to "directio" everything was fine and I haven't gotten any corruption again. Aviad

JRE Plug-in “Next-Generation” – Part II

In my last post "JRE Plug-in “Next-Generation” – to migrate or not?" I wrote about a Forms launching issue in EBS right after upgrading JRE (Java Plug-in) to version 6 update 11 which works with the new next-generation Java Plug-in architecture. The problem happens inconsistently and it only works when I disable the "next-generation Java Plug-in". Following a SR I've opened to Oracle support about this issue, I was being asked to verify that the profile option "Self Service Personal Home Page Mode" is set to "Framework Only". We have this profile option set to "Personal Home Page" as our users prefer this way to the "Framework Only" way. It's important to note that "Personal Home Page" is not a supported value for the "Self Service Personal Home Page Mode" profile option and may cause unexpected issues. After setting the profile option to "Framework Only" the problem has resolved and the screen doesn't freezes anymore. So the solution in my case was to set the profile option "Self Service Personal Home Page Mode" to "Framework Only" (we are still testing it but it look fine so far), however there are two more options that seems to work, even when the profile option set to "Personal Home Page" and "next generation Java Plug-in" is enabled. 1) Uncheck "Keep temporary files on my computer" - Navigate to Java console (start -> settings -> Control Panel -> Java,  or start -> run -> javacpl.cpl) - On General tab -> Temporary Internet Files -> Settings -> uncheck the "Keep temporary files on my computer". - Check the issue from a fresh IE session.   I'm not sure how or why, but it solves the problem, no more freezing this way.. 2) Set “splashScreen” to null - Edit $FORMS60_WEB_CONFIG_FILE file in your Forms server node. - Change this line "splashScreen=oracle/apps/media/splash.gif" to "splashScreen=" - No need to bounce any service. - Check the issue from a fresh IE session. Again, it's not so clear how or why, but it solves the problem as well. Now, we just need to convince our users to accept the "framework only" look and feel, and then we would consider upgrading all our clients to the new next-generation Java Plug-in. You are welcome to leave a comment or share your experience with the new Java Plug-in. Aviad

JRE Plug-in “Next-Generation” – to migrate or not?

It has been more than half a year since we've migrated from Oracle Jinitiator to Sun Java JRE Plug-in (Java 6 update 5) in our Oracle Applications (EBS) system, and I must say, I'm not satisfied yet. For the first months we had been struggling with a lot of mouse focus bugs which have made our users very angry about this upgrade. Although we've applied some patches related to this bugs, we still have some with no resolution. Upgrading to Developer 6i patchset 19 has solved some bugs but not all of them. As part of an SR we had opened about mouse focus issue, we was advised by Oracle to install the latest Java JRE (Java 6 update 12 this days) as a possible solution for the remaining bugs. Starting with Java 6 update 10, Sun has introduced the new "next-generation Java Plug-in", which makes troubles with Oracle EBS. You can read more about this new architecture at Sun Java site - "What is next-generation Java Plug-in". Right after installing Java 6 update 11, I encountered a problem - when trying to open forms the screen freezes. The browser window hangs inconsistently. I have no idea when it's going to be opened and when it's not. I've tried Java 6 update 12 and it's the same – sometimes it opens and sometimes it doesn’t. No matter what I did - clear java cache on client, clear Apache cache, install JRE in different directory (in case you have installed previous update of version 6), uninstall previous versions of Java Plug-in installed on same pc, I tried with explorer 6 and 7 - the problem wasn't resolved. There is an unpublished opened bug for this problem: Bug 7875493 - "Application freezes intermittently when using JRE 6U10 and later". I've been told by Oracle support that they have some incompatibilities with the new next-generation architecture and that they are working with Sun about it. Meanwhile there are 2 workarounds: (the second doesn't work for me but suggested by Oracle support) 1) Disable the "next generation Java Plug-in" option: Go to Control Panel -> Java -> Select the "Advanced" tab -> expand the "Java Plug-in" -> uncheck the "Enable the next-generation Java Plug-in" option.   This workaround always works (at least for me...). 2) Set the swap file to system managed + Tune the heap size for java: - Go to Control Panel -> System -> Select the "Advanced" tab -> click on Settings (in Performance frame) -> Select the "Advanced" tab -> Click on Change -> Select the "System managed size" option. - Go to Control Panel -> Java -> Select the "Java" tab -> Click "View..." (in Java Applet Runtime Settings frame) -> update the "Java Runtime Parameters" field with: "-Xmx128m -Xms64m". This workaround doesn't work for me. For now, I've decided to stay with the "old" Java Plug-in 6 update 5 and do not upgrade our users to the new next-generation Java Plug-in. I Hope the following updates of Java Plug-in will be better or Oracle will publish a patch to solve this problem. I’ll keep update as soon as I have more info’. Aviad

How to enable trace for a CRM session

I was being asked to examine a performance issue within one of our CRM application screens, after some users complained about a specific long time action. First thing, I tried to enable trace for the CRM session, but It turned out that it’s definitely not simple to identify a CRM session. Especially in my case, when a session opens two (sometimes more) database sessions. It’s quite impossible actually. So how it is possible to trace those CRM sessions anyway? Oracle has provided an option to execute custom code for every session opened in the database through a system profile. This profile called “Initialization SQL Statement - Custom” (the short name is 'FND_INIT_SQL') and allows customize sql/pl*sql code. Once setting this profile in user level, each session opened for this user will first execute the code within the profile. No matter which type of activity the user does – Forms, CRM, Concurrent request, or anything else that opens a database session – the content of this profile will be executed. So, clearly we can use this capability to enable Trace for users sessions. Steps to enable trace for specific user: Login with “Application Developer” responsibility Open the “Create Profile” form –> Query the profile “FND_INIT_SQL” Make sure that “visible” and “updateable” are checked in user level.   Switch responsibility to “System Administrator” Navigate to Profile –> System –> Query the profile “Initialization SQL Statement - Custom” in user level for the user we would like to enable trace for.   Update the profile option value in user level to the following: BEGIN FND_CTL.FND_SESS_CTL('','', '', 'TRUE','','ALTER SESSION SET TRACEFILE_IDENTIFIER='||''''||'AVIADE' ||''''||' EVENTS ='||''''||' 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12 '||''''); END;     ** Just replace AVIADE with the user you enable trace for.    Now, after the user logout from the application (the user you enabled trace for), the user can login and reproduce the issue.   When finish to reproduce the issue, you should disable the trace by clearing the profile option value and update it to NULL. (profile “Initialization SQL Statement – Custom” of course..) The trace file/s will wait for you in your udump (user_dump_dest init’ parameter) directory. Since I enabled and disabled the trace quite a few times while investigating my performance issue, I wrote these handy simple programs which enable and disable the trace for a user in a quick and easy manner. Execute this program to enable trace for a specific user: (substitute step 6 above) DECLARE   l_ret     boolean;   l_user_id number; BEGIN   select user_id     into l_user_id     from fnd_user    where user_name = '&&USER_NAME';   l_ret := fnd_profile.SAVE(X_NAME        => 'FND_INIT_SQL',                             X_VALUE       => 'BEGIN FND_CTL.FND_SESS_CTL('''','''','''', ''TRUE'','''',''ALTER SESSION SET TRACEFILE_IDENTIFIER=''||''''''''||''&&USER_NAME'' ||''''''''||'' EVENTS =''||''''''''||'' 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12 ''||''''''''); END;',                             X_LEVEL_NAME  => 'USER',                             X_LEVEL_VALUE => l_user_id);   commit;   dbms_output.put_line('Profile has updated successfully'); EXCEPTION   when others then     dbms_output.put_line('Failed to update the profile: '||sqlerrm); END;   Execute this program to disable trace for a specific user: (substitute step 8 above) DECLARE   l_ret     boolean;   l_user_id number; BEGIN   select user_id     into l_user_id     from fnd_user    where user_name = '&USER_NAME';   l_ret := fnd_profile.DELETE(X_NAME        => 'FND_INIT_SQL',                               X_LEVEL_NAME  => 'USER',                               X_LEVEL_VALUE => l_user_id);   commit;   dbms_output.put_line('Profile has erased successfully'); EXCEPTION   when others then     dbms_output.put_line('Failed to erase the profile: '||sqlerrm); END; Hope you find it helpful… Feel free to leave a comment or share your thought about this issue. Aviad

How to Configure OEM 10g Database Control after Clone

After cloning an EBS environment I had tried to start the OEM 10g Database Control (DBConsole) and I got this message: [oratest@testsrv1 ~]$ emctl start dbconsole TZ set to Israel OC4J Configuration issue. /oracle/test/db/testdb/10.2/ oc4j/j2ee/OC4J_DBConsole_testsrv1.domain_TEST not found. However, it was an acceptable message since DB Console wasn't configured to this instance yet. Using emca I've tried to recreate the DB Control configuration files as follow: [oratest@testsrv1 ~]$ emca -config dbcontrol db ... INFO: Database Control started successfully ... INFO: >>>>>>>>>>> The Database Control URL is http://testsrv1.domain:5500/em <<<<<<<<<<< Enterprise Manager configuration completed successfully Well, I thought it looks fine, but when I opened the DB Console I saw this message at the top of the page:  error: java.lang.Exception: Exception in sending Request :: null After some research, logs digging and Metalink searches, I found that the Agent component should be relinked manually. But before there are two issues in env_sysman.mk file that should be fixed before executing the relink command. I will show my actions step by step: Stop DB Console: emctl stop dbconsole   cd $ORACLE_HOME/sysman/lib   Edit env_sysman.mk and look for "JRE_LIB_DIR" In my instance JRE_LIB_DIR was populated with the old environment name.. I changed it from: JRE_LIB_DIR=/oracle/prod/db/proddb/10.2/jdk/jre/lib/i386 To: JRE_LIB_DIR=/oracle/test/db/testdb/10.2/jdk/jre/lib/i386   One more issue for env_sysman.mk, look for "$(LIBNMEM)  $(LIBNMER)" And change from: $(LIBNMEM)  $(LIBNMER) \ To: $(LIBNMEM)  $(LIBNMER) $(LIBNMO) \   Now relink the Agent component by issuing this command: make -f ins_sysman.mk agent   Recreate the DB Control configuration files (Again, even if you did it before) emca -config dbcontrol db Following this procedure will make the DBConsole configured and work for the new cloned instance.

How to enable FND_HELP search by creating an index on FND_LOBS efficiently

Have you ever tried to use the search option of Online Help in Oracle Applications? Our users did... and they got "The page cannot be found" message... I checked it on firefox, hope to get more accurate message, and I got this: "Not Found. The requested URL /pls/DEV/fnd_help.search was not found on this server".     I checked fnd_help package and it's compiled and looks fine. These errors appeared in error_log_pls: [Mon Jul 28 10:34:54 2008] [warn] mod_plsql: Stale Connection due to Oracle error 20000 [Mon Jul 28 10:34:54 2008] [error] mod_plsql: /pls/DEV/fnd_help.search ORA-20000 ORA-20000: Oracle Text error: DRG-10599: column is not indexed ORA-06512: at "SYS.DBMS_SYS_SQL", line 1209 ORA-06512: at "SYS.DBMS_SQL", line 328 ORA-06512: at "APPS.FND_HELP", line 1043 ORA-06512: at "APPS.FND_HELP", line 873 ORA-06512: at line 20 I found note 306239.1 - "Cannot Search Online Help After Fresh Install of 11.5.10" which suggest that Applications interMedia Text indexes are corrupt. In my instance it even wasn't exist... I followed this note which instructs to: drop index FND_LOBS_CTX; Rebuild the index using aflobbld.sql But aflobbld.sql had been running for more than 10 hours and the size of DR$FND_LOBS_CTX$I table has reached to 35 GB ! I had been wondering how it can be that fnd_lobs table is less than 1GB and the index on it is 35 GB and counting.... ?! Note 396803.1 - "FND_LOBS_CTX is having huge size, how to reduce the sizeof the index?" suggests it's a bug, indexing all documents in FND_LOB table, also the binary files, while using wrong filter. So how can we make aflobbld.sql to index only FND_HELP documents? For each row in FND_LOBS table the file_format column is populated with one of the following values: IGNORE, BINARY, TEXT. aflobbld.sql will index only rows that have this column set to BINARY or TEXT. If we set all rows to IGNORE except FND_HELP rows, we could index them only. Note 397757.1 - "How to Speed Up Index Creation on FND_LOBS by indexing Only FND_HELP Data" suggests the steps to do it. These are the steps: Backup the fnd_lobs table before updating it, we will use it later:   create table fnd_lobs_bk as select * from fnd_lobs;   ** you can create a backup of this table and omit the file_date column to make this backup faster   Drop index FND_LOBS_CTX if exists:   drop index applsys.FND_LOBS_CTX;   Update all rows to IGNORE except FND_HELP rows:   update fnd_lobs set file_format = 'IGNORE' where nvl(program_name,'@') <> 'FND_HELP' ;   Execute aflobbld.sql from OS terminal:   sqlplus apps/sppas @$FND_TOP/sql/aflobbld.sql applsys apps;   Since I'm not sure about the impact of leaving the FND_LOBS rows as IGNORE, I updated them back to the previous state:   create unique index fnd_lobs_bk_u1 on fnd_lobs_bk (file_id);   update (select fl.file_format ffo,flb.file_format ffb         from fnd_lobs fl             ,fnd_lobs_bk flb         where fl.file_id = flb.file_id) set ffo=ffb;   drop table fnd_lobs_bk;   Check the search option.... it should work now. You are welcome to leave a comment . Aviad

How to execute TKPROF on trace files larger than 2GB ? --> Use pipe

Here is a nice trick to work with files larger than 2GB on Unix/Linux using pipe. First case - TKPROF When trying to execute TKPROF on a trace file larger than 2 GB I got this error: [udump]$ ll test_ora_21769.trc -rw-r-----  1 oratest dba 2736108204 Jun 23 11:04 test_ora_21769.trc [udump]$ tkprof test_ora_21769.trc test_ora_21769.out TKPROF: Release 9.2.0.6.0 - Production on Thu Jun 23 21:05:10 2008 Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved. could not open trace file test_ora_21769.trc In order to successfully execute TKPROF on this trace file you can use the mkfifo command to create named pipe as follow: Open a new unix/linux session (1st), change directory where the trace file exists and execute: [udump]$ mkfifo mytracepipe [udump]$ tkprof mytracepipe test_ora_21769.out TKPROF: Release 9.2.0.6.0 - Production on Thu Jun 23 21:07:35 2008 Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved. Open another session (2nd), change directory where the trace file exists and execute: [udump]$ cat test_ora_21769.trc > mytracepipe This way you'll successfully get the output file.   Second case - spool Similar issue with spool to file larger than 2GB can be treat similarly. $ mkfifo myspoolpipe.out --> Create new named pipe called 'myspoolpipe.out' $ dd if=myspoolpipe.out of=aviad.out & --> What you read from 'myspoolpipe.out' write to 'aviad.out' $ sqlplus user/pwd@dbname SQL*Plus: Release 9.2.0.6.0 - Production on Tue Jun 24 12:05:37 2008 Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved. Connected to: Oracle9i Enterprise Edition Release 9.2.0.6.0 - Production With the Partitioning, OLAP and Oracle Data Mining options JServer Release 9.2.0.6.0 - Production SQL> spool myspoolpipe.out --> Spool to the pipe SQL> select ..... SQL> spool off SQL> 5225309+294082 records in 5367174+1 records out SQL> exit Disconnected from Oracle9i Enterprise Edition Release 9.2.0.6.0 - Production With the Partitioning, OLAP and Oracle Data Mining options JServer Release 9.2.0.6.0 - Production [1]+  Done                    dd if=myspoolpipe.out of=aviad.out $ ls -ltr prw-r--r--  1 oratest dba          0 Jun 24 12:22 myspoolpipe.out -rw-r--r--  1 oratest dba 2747993487 Jun 24 12:22 aviad.out Related Notes: Note 62427.1 - 2Gb or Not 2Gb - File limits in Oracle Note 94486.1 - How to Create a SQL*Plus Spool File Larger Than 2 GB on UNIX Aviad

Mix of Old & New style buttons in OA Framework pages

After some heavy patches applied on our system we noticed that some buttons in OAF pages looks like the old style gray buttons while the others are fine new style yellow buttons. For example:   (The "Advanced" is the old style and all the others are the new style) Trying to clear cache ($COMMON_TOP/_pages) and bounce Apache didn't solve the problem. The solution is hiding within jserv.properties: Edit $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties Change the following to TRUE: wrapper.bin.parameters=-Djava.awt.headless=true (optional) Clear all content from $OA_HTML/cabo/images/cache (e.g rm -rf $OA_HTML/cabo/images/cache) (optional) Clear all content from $COMMON_TOP/_pages Bounce Apache And the problem will be resolved...   In order to make this change permanent, you should update the Application context file as follow, otherwise next run of AutoConfig will overwrite your change. Edit $APPL_TOP/admin/$CONTEXT_NAME.xml Change the following to: <java_awt_headless oa_var="s_java_awt_headless">true</java_awt_headless> Run AutoConfig on Apps Tier. Bounce Apache Related Note: 368188.1 - Buttons Are Not Rendering Correctly In Self Service Framework Pages. Aviad

FND_GLOBAL affected by New Global Performance Changes

After applying ATG Rollup 5 patch (and above) we discovered an issue with some of our custom developments. For some processes we got the following errors: ORA-20001: Oracle error -20001: ORA-20001: Oracle error -4092: ORA-04092: cannot SET NLS in a trigger has been detected in fnd_global.set_nls.set_parameter('NLS_LANGUAGE','AMERICAN'). has been detected in fnd_global.set_nls. ORA-06512: at "APPS.APP_EXCEPTION", line 72 ORA-06512: at "APPS.FND_GLOBAL", line 240 ORA-06512: at "APPS.FND_GLOBAL", line 1410 ORA-06512: at "APPS.FND_GLOBAL", line 1655 ORA-06512: at "APPS.FND_GLOBAL", line 2170 ORA-06512: at "APPS.FND_GLOBAL", line 2312 ORA-06512: at "APPS.FND_GLOBAL", line 2250 and this: ORA-20001: Oracle error -2074: ORA-02074: cannot SET NLS in a distributed transaction has been detected in fnd_global.set_nls.set_paramenters('NLS_LANGUAGE','AMERICAN'). After some debug work we found that this issue happens when executing FND_GLOBAL.apps_initialize more than once within a trigger/via a db link in the same transaction. According to Note: 556391.1 - "ORA-02074: Cannot SET NLS in a Distributed Transaction" this issue cause by a new global performance changes. Oracle Development said: "Very sorry if the new global performance changes have exposed you to this error, but there is no way we can back out these changes. They are not only complex and wide spread but required to maintain functional performance levels. Using fnd_global to change user/resp context from a trigger is not only not supported it is ill advised." OK, So we had to find a workaround to this issues and we found two... I'll start with a sample of the new behavior of fnd_global to demonstrate the issue and the solutions/workarounds will come right after. SQL> create table test1 (a number, b number); Table created SQL> insert into test1 (a) values (1001); 1 row inserted SQL> insert into test1 (a) values (1002); 1 row inserted SQL> commit; Commit complete SQL> create or replace trigger test1_trg_bi   2  after update on test1   3  for each row   4  begin   5       fnd_global.APPS_INITIALIZE(:new.a,1,1);   6       -- fnd_request.submit_request...   7       -- ....   8       -- ....   9  end; 10  / Trigger created SQL> select fnd_global.user_id from dual;     USER_ID ----------         -1 SQL> update test1 set b=1101 where a=1001; 1 row updated SQL> select fnd_global.user_id from dual;     USER_ID ----------       1001 SQL> update test1 set b=1102 where a=1002; update test1 set b=1102 where a=1002 ORA-20001: Oracle error -20001: ORA-20001: Oracle error -4092: ORA-04092: cannot SET NLS in a trigger has been detected in fnd_global.set_nls.set_parameter('NLS_LANGUAGE','AMERICAN'). has been detected in fnd_global.set_nls. ORA-06512: at "APPS.APP_EXCEPTION", line 72 ORA-06512: at "APPS.FND_GLOBAL", line 240 ORA-06512: at "APPS.FND_GLOBAL", line 1410 ORA-06512: at "APPS.FND_GLOBAL", line 1655 ORA-06512: at "APPS.FND_GLOBAL", line 2170 ORA-06512: at "APPS.FND_GLOBAL", line 2312 ORA-06512: at "APPS.FND_GLOBAL", line 2250 ORA-06512: at "APPS.TEST1_TRG_BI", line 2 ORA-04088: error during execution of trigger 'APPS.TEST1_TRG_BI' As you can see, the second update failed because apps_initialize was executed for the second time in the same transaction. Now I'll show two ways to workaround this issue: 1) As suggested in Note: 556391.1 - "ORA-02074: Cannot SET NLS in a Distributed Transaction" a wrapper Concurrent Request which contain a call to the context set (apps_initialize) and afterwards submits the original request, is one possible solution. instead: create or replace trigger test1_trg_bi after update on test1 for each row declare      . . . begin      fnd_global.APPS_INITIALIZE(:new.a,1,1);      ret_code := fnd_request.submit_request ('OWNER', 'ORIGINAL_CONC', . . .);      . . .      . . . end; create the following trigger: create or replace trigger test1_trg_bi after update on test1 for each row declare      . . . begin      ret_code := fnd_request.submit_request ('OWNER', 'WRAPPER_CONC', . . . , :new.a, . . . );      . . .      . . . end; additionally - create a new plsql concurrent (WRAPPER_CONC) that contains the fnd_global.apps_initialize and submits the ORIGINAL_CONC concurrent request. This way, the apps_initialize statement executed in a separate transaction with no error. This is the preferred and recommended solution by Oracle. 2) The second solution is easier to implement, works fine but according to Note: 556391.1 is not supported since it contains calls to fnd_global within a database trigger. Anyway... The idea is to call the apps_initialize in an Autonomous Transaction procedure. Follow this sample: SQL> create or replace procedure test1_apps_init (p_user_id number) is   2  pragma autonomous_transaction;   3  begin   4       fnd_global.APPS_INITIALIZE(p_user_id,1,1);   5       commit;   6  end;   7  / Procedure created SQL> create or replace trigger test1_trg_bi   2  after update on test1   3  for each row   4  begin   5       test1_apps_init (:new.a);   6       -- fnd_request.submit_request...   7       -- ....   8       -- .....   9  end; 10  / Trigger created SQL> select fnd_global.user_id from dual;     USER_ID ----------         -1 SQL> update test1 set b=1101 where a=1001; 1 row updated SQL> select fnd_global.user_id from dual;     USER_ID ----------       1001 SQL> update test1 set b=1102 where a=1002; 1 row updated SQL> select fnd_global.user_id from dual;     USER_ID ----------       1002 As you can see, the update statements were executed successfully this time and the session was updated with the appropriate user context in each update statement. Those two solutions are working fine, but keep in mind that the second is not supported. You are welcome to leave a comment. Aviad

UTL_FILE_DIR issue after applying patch 5985992 'TXK AUTOCONFIG RUP Q'

Do you have more than 240 characters in your utl_file_dir? if so, you should read this before applying patch 5985992 'TXK AUTOCONFIG RUP Q (Jul/Aug 2007)'. After applying this patch, AutoConfig on database tier failed with the following error: [PROFILE PHASE]   AutoConfig could not successfully execute the following scripts:     Directory: [RDBMS_ORACLE_HOME]/appsutil/install/[context_name]        afdbprf.sh              INSTE8_PRF         1 AutoConfig is exiting with status 1 It wasn't clear why afdbprf.sh failed so I ran it manually from ssh terminal and I got this: $ [RDBMS_ORACLE_HOME]/appsutil/install/[context_name]/afdbprf.sh afdbprf.sh started at Tue Jan 29 17:43:21 IST 2008 The environment settings are as follows ...        ORACLE_HOME : ....        .........        ......... Application Id : 0 Profile Name : BIS_DEBUG_LOG_DIRECTORY Level Id : 10001 New Value : /usr/tmp Old Value : /usr/tmp declare * ERROR at line 1: ORA-12899: value too large for column "APPLSYS"."FND_PROFILE_OPTION_VALUES"."PROFILE_OPTION_VALUE" (actual: 486, maximum: 240) ORA-06512: at line 44 ORA-06512: at line 139 Looking into afdbprf.sql  (executed by afdbprf.sh) reveals the problem: ....... --Setting  BIS_DEBUG_LOG_DIRECTORY            set_profile(0, 'BIS_DEBUG_LOG_DIRECTORY',                        10001, 0,                        '&BIS_DEBUG_LOG_DIR',                        NULL); -- This profile option was earlier set in addbprf.sql via bug 2843457, Now moved here for bug 5722442 -- -- Set up UTL_FILE_LOG profile option --           set_profile(1, 'UTL_FILE_LOG',                       10001, 0, '/usr/tmp,........[more than 240 characters..... :-) ]',                       NULL); In order to fix bug 5722442, the UTL_FILE_LOG updated with the value of s_db_util_filedir from the database context file and a new bug was created (not a bug according to Bug 6404909). If the s_db_util_filedir contains more than 240 characters it can't be stored in a profile since the PROFILE_OPTION_VALUE column defined as varchar2(240). The s_db_util_filedir initialized when creating the database context file by adbldxml.pl, and set up with the value of utl_file_dir database parameter. I'm not sure why it should be updated with all directories within utl_file_dir and not with the relevant directories only...?!  The UTL_FILE_LOG profile (or "Stored procedure log directory") wasn't updated with AutoConfig before applying this patch, so I'm not sure regarding the purpose of this profile. Before applying this patch I have this profile set to a directory that doesn't exist... The solution according to Note:458511.1 - "After patch 5985992 AutoConfig On Database Tier Fails with script afdbprf.sh" is to change the value of s_db_util_filedir in the database context file to a value less the 240 characters and run AutoConfig again. And it works, of course... What should I do if I my utl_file_dir contains more than 240 characters?! Actually I don't have a good answer to this question but I will try to locate all the places on which this issue might affect when s_db_util_filedir will be updated with a value less than 240 chars. AutoConfig on the DB Tier creates the [SID]_APPS_BASE.ora file under $RDBMS_ORACLE_HOME/dbs (if it's not already exist), which contains the utl_file_dir database parameter generated respectively to the s_db_util_filedir from the database context file. So if you plan to rebuild your init.ora with AutoConfig you will need to update the utl_file_dir manually. When executing adbldxml.pl on DB Tier to rebuild the database context file, the s_db_util_filedir will be filled with the utl_file_dir database parameter - that might contain more than 240 chars. Therefore, before executing AutoConfig you should edit the new context file and shorten the value of s_db_util_filedir. The UTL_FILE_LOG profile ("Stored procedure log directory")  will be updated with s_db_util_filedir shortened value. You are welcome to leave a comment or update with additional information regarding this issue. Aviad

Oracle Discoverer Query Prediction functionality and Performance

Lately we noticed that our Discoverer reports runs very slow. Actually, the problem wasn’t the Discoverer reports query but the query prediction that for some reason took so long. What is Query Prediction in Discoverer? “Discoverer includes functionality to predict the time required to retrieve the information in a Discoverer query. The query prediction appears before the query begins, enabling Discoverer users to decide whether or not to run the query. This is a powerful facility that enables Discoverer users to control how long they wait for large reports.” (from Oracle doc’) The query prediction is the elapsed time while the following message appeared on the bottom left of Discoverer Desktop window: "Determining query time estimate". For each report we tested, we found query prediction runs 30%-50% (!!!) from the report’s total run time. Next phase was to start a SQL trace on Discoverer session to see what actually happens when running a Discoverer report. This is the relevant section from the SQL Trace: SELECT QS_ID, QS_COST, QS_ACT_CPU_TIME,        QS_ACT_ELAP_TIME, QS_EST_ELAP_TIME,        QS_CREATED_DATE, QS_OBJECT_USE_KEY,        QS_NUM_ROWS,QS_STATE FROM [EUL_USER].EUL5_QPP_STATS WHERE  QS_COST IS NOT NULL AND    QS_OBJECT_USE_KEY = :OBJECTUSEKEY ORDER BY QS_CREATED_DATE DESC As you can see, the query prediction functionality tries to retrieve statistics information from EUL5_QPP_STATS and it takes 35 seconds. (total time for this report is 55 seconds). The query prediction based, among other things, on the query prediction statistics table – EUL5_QPP_STATS. This table records query prediction statistics while running Discoverer reports. There is no logic by estimating query run time longer than the report’s query itself… Since the query prediction functionality is important to our users we avoid from disable this functionality (by setting the QPPEnable to 0). Furthermore, I found that we have statistics data in this table from 7 years ago…  There is no meaning to hold these statistics… I tried to find information about purging the EUL5_QPP_STATS and I found this: “How to delete old query prediction statistics” in Oracle® Business Intelligence Discoverer Administration Guide 10g Release 2 (10.1.2.1) There is a SQL script at [ORACLE_ HOME]\discoverer\util\eulstdel.sql – that deletes all query prediction statistics that were created before a specified date. Great! I executed this sql on my database, gave 90 days as a parameter and it deleted 460,000 (from 468,000) rows. I ran a Discoverer report again, but still query prediction takes too long, same like before. I checked the explain plan and the cost of the above SQL and it remains the same. I tried to gather statistics on EUL5_QPP_ENABLE table and rebuild its indexes but cost become higher… (More than 103, something like 800…). I had no choice but rebuild the EUL5_QPP_ENABLE table (by export, drop table and import). After recreation of EUL5_QPP_STATS table I ran a Discoverer report again and query prediction takes insignificant time, almost nothing…  :-) This is from the trace I took after: SELECT QS_ID, QS_COST, QS_ACT_CPU_TIME,        QS_ACT_ELAP_TIME, QS_EST_ELAP_TIME,        QS_CREATED_DATE, QS_OBJECT_USE_KEY,        QS_NUM_ROWS,QS_STATE FROM [EUL_USER].EUL5_QPP_STATS WHERE  QS_COST IS NOT NULL AND    QS_OBJECT_USE_KEY = :OBJECTUSEKEY ORDER BY QS_CREATED_DATE DESC The elapsed time for this sql reduced to 0.05 seconds!! (was 35 sec’ before) Sql cost reduced from 103 to 31! I checked this issue on Discoverer Desktop 10g (10.1.2.2) but it is relevant to the web tools (Discoverer viewer and Discoverer Plus) as well, since the query prediction functionality exist in these tools like in the client version. You are welcome to leave a comment. Aviad

Troubleshooting after Upgrade Oracle Applications 11i to 10g

Following the last post on Troubleshooting during Upgrade Oracle Applications 11i to 10g, in this post I'm going to write about troubleshooting AFTER upgrade Oracle Applications 11i to 10g. Case 1: ORA-29877: FAILED IN THE EXECUTION OF THE ODCIINDEXPDATE ROUTINE Symptom: When trying to save in "Find Service Request" Form, an error message shows up: ORA-29877: failed in the execution of the ODCIINDEXUPDATE routine ORA-20000: Oracle Text error: DBG-50857: oracle error in textindexmethods. ODCIIndexUpdate ORA-20000: Oracle Text error: DBG-10602: failed to queue DML change to column for in Package CS_ServiceRequest_PVT Procedure Update_ServiceRequest Cause: KOREAN_LEXER was deprecated in release 9.0.1 and desupported in release 10.2.0.1. It is replaced by KOREAN_MORPH_LEXER and Oracle Text indexes need to be migrated from the obsolete KOREAN_LEXER to the new KOREAN_MORPH_LEXER. This was not done during the database upgrade. Solution: Run the following script as sysdba: sqlplus "/ as sysdba" @$ORACLE_HOME/ctx/sample/script/drkorean.sql If you get this error (or similar): $ sqlplus "/as sysdba" @drkorean.sql SQL*Plus: Release 10.2.0.3.0 - Production on … Copyright (c) 1982, 2006, Oracle. All Rights Reserved. Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Production With the Partitioning, OLAP and Data Mining options List of indexes that use KOREAN_LEXER as top level lexer: List of indexes that use KOREAN_LEXER as a sub lexer: "CS"."CS_FORUM_MESSAGES_TL_N4" "CS"."CS_INCIDENTS_ALL_TL_N1" "CS"."CS_KB_SOLN_CAT_TL_N1" "CS"."SUMMARY_CTX_INDEX" "ICX"."ICX_QUES_CTX" Migrate KOREAN_LEXER to KOREAN_MORPH_LEXER Rebuild all indexes that use korean lexer as top level lexer: Reindex all documents that use KOREAN_LEXER as sub lexer reindexing : "CS"."CS_FORUM_MESSAGES_TL" finished. reindexing : "CS"."CS_INCIDENTS_ALL_TL" finished. declare * ERROR at line 1: ORA-29861: domain index is marked LOADING/FAILED/UNUSABLE ORA-06512: at line 316 It is mean that one or more Text indexes are invalid so we have to check which indexes are affected. Run this query to check which indexes are invalid: select owner, index_name, index_type, status, domidx_status, domidx_opstatus from dba_indexes where ityp_owner='CTXSYS' order by 1; select idx_owner, idx_name, idx_type, idx_status from ctxsys.ctx_indexes order by 1; Drop the affected indexes (e.g. drop index cs.CS_KB_SOLN_CAT_TL_N1 force;) Recreate dropped indexes as described in Metalink Note 312640.1 - Oracle Text: Re-installation of Applications 11i (11.5.10) Oracle Text Indexes Section 2: Create the CS Text index preferences and indexes (7, 8, 9, 10 & 11) Rerun script $ORACLE_HOME/ctx/sample/script/drkorean.sql to complete the lexer conversion. Case 2: FRM-40735: ON-ERROR TRIGGER ORA-01001 FRM-42100 WHEN SEARCHING PARTIALLY PEOPLE Symptom: In Users form, when trying to query by person field like ‘%name%’ an error message shows up: FRM-40735: ON-ERROR TRIGGER RAISED UNHANDLED EXCEPTION ORA-01001 At alert_SID.log: ORA-07445: exception encountered: core dump [qerixGetKey()+562] [SIGSEGV] [Address not mapped to object] [0x10] [] [] Cause: Initialization parameters didn’t set properly for 10g. Solution: Add the follwing to initSID.ora file: _b_tree_bitmap_plans = FALSE For additional information see note 216205.1 - Database Initialization Parameters for Oracle Applications 11i Case 3: Error in iAS Cache: java.io.NotSerializableException: oracle.apps.qot.util.LookupValuesKey Symptom: When trying to access the Quote tab from sales online, an error message is returned: Error in iAS Cache: java.io.NotSerializableException: oracle.apps.qot.util.LookupValuesKey Restart Apache resolve the issue for a short time but after some clicks it’s happens again. Solution: Should apply patch 6024690. Note 422499.1 - LookupValuesKey Error Accessing Quote Tab Case 4: Oracle Forms Web CGI: Error detected Symptom: After login to applications and choose a responsibility, just before java applet startup, an error message shows up: Oracle Forms Web CGI: Error detected Your request cannot be serviced at this time due to the following error: ERROR: URL Contains One or More Restricted Characters Solution: 1) Comment out the value FORMS60_BLOCK_URL_CHARACTERS in apps.conf. 2) Bounce Apache. Follow Note:395628.1 - 11.5.10 Error: URL Contains One Or More Restricted Characters You are welcom to leave a comment... Aviad

Troubleshooting during Upgrade Oracle Applications 11i to 10g

During the Oracle Apps Database upgrade to 10g that I wrote about in the previous post, I ran into some errors. In this post I will write about the errors symptoms, causes and the solutions that worked for me. :-) Case 1: XDB (Oracle XML Database) schema is invalid Symptom: During the upgrade process an error message shows up: ORA-06576: not a valid function or procedure name In Oracle_Server.log: ($ORACLE_HOME/cfgtoollogs/dbua/[SID]/upgrade/) call dbms_xdb.CleanSGAForUpgrade(); call dbms_xdb.CleanSGAForUpgrade(); * ERROR at line 1: ORA-06576: not a valid function or procedure name When click on “Ignore” an error message shows up: ORA-31159: XML DB is in an invalid state ORA-00600: internal error code, arguments: [qmtb_init_len], [], [], [], [], [], Cause: Oracle XML Database (XDB) is in an Invalid state. Execute this query to check the XDB state: select comp_name, status, version from DBA_REGISTRY where comp_name= 'Oracle XML Database’; COMP_NAME STATUS VERSION Oracle XML Database INVALID 9.2.0.6.0 Solution: The XDB schema should be reinstalled. I’ve Followed Note 243554.1 - How to Deinstall and Reinstall XML Database (XDB) Step by Step Action Plan: 1) Restart the DB 2) Connect to DB as sysdba 3) Run $ORACLE_HOME/rdbms/admin/catnoqm.sql to remove XDB from database. 4) Update init.ora with the following (at least): Shared_pool_size=150MB Java_pool_size=150MB 5) Shutdown immediate 6) Startup 7) Run $ORACLE_HOME/rdbms/admin/catqm.sql XDB XDB TEMP to create the XDB schema with all required objects 8) Run $ORACLE_HOME/rdbms/admin/catxdbj.sql to register XDB java api jar files. 9) Check invalid objects for xdb users select count(*) from dba_objects where owner='XDB' and status='INVALID'; 10) Check dba_registry (should be valid now…) select comp_name, status, version from DBA_REGISTRY where comp_name= 'Oracle XML Database’; Case 2: ORA-06512 AT SYS.UTL_RECOMP Symptoms: During the upgrade process, an error message shows up: ORA-01003: no statement parsed ORA-06512: at "SYS.UTL_RECOMP", line 464 ORA-06512: at "SYS.UTL_RECOMP", line 467 ORA-06512: at "SYS.UTL_RECOMP", line 607 ORA-01003: no statement parsed ORA-06512: at line 4 I’ve take a look on log file $ORACLE_HOME/cfgtoollogs/dbua/[SID]/upgrade/PostUpgrade.log: DECLARE 2 threads pls_integer := &&1; 3 BEGIN 4 utl_recomp.recomp_parallel(threads); 5 END; 6 / DECLARE * ERROR at line 1: ORA-01003: no statement parsed ORA-06512: at "SYS.UTL_RECOMP", line 464 ORA-06512: at "SYS.UTL_RECOMP", line 467 ORA-06512: at "SYS.UTL_RECOMP", line 607 ORA-01003: no statement parsed ORA-06512: at line 4 And those are the last lines of the related trace file: [Thread-23] [18:12:10:580] [SQLEngine.setSpool:1744] Setting Spool = /oracle/tst10db/10.2/cfgtoollogs/dbua/TST10/upgrade/PostUpgrade.log [Thread-23] [18:12:10:581] [SQLEngine.setSpool:1745] Is spool appendable? --> true [Thread-23] [18:12:10:584] [SQLEngine.reInitialize:602] Reinitializing SQLEngine... [Thread-23] [18:12:10:591] [SQLEngine.initialize:241] Execing SQLPLUS/SVRMGR process... [Thread-23] [18:12:10:609] [SQLEngine.initialize:269] m_bReaderStarted: false [Thread-23] [18:12:10:610] [SQLEngine.initialize:273] Starting Reader Thread... [Thread-23] [18:12:10:617] [SQLEngine.initialize:322] Waiting for m_bReaderStarted to be true [Thread-57] [18:58:44:357] [BasicStep.handleNonIgnorableError:430] oracle.sysman.assistants.util.UIMessageHandler@c8e4de:messageHandler [Thread-57] [18:58:44:361] [BasicStep.handleNonIgnorableError:431] ORA-01003: no statement parsed ORA-06512: at "SYS.UTL_RECOMP", line 464 ORA-06512: at "SYS.UTL_RECOMP", line 467 ORA-06512: at "SYS.UTL_RECOMP", line 607 ORA-01003: no statement parsed ORA-06512: at line 4 Cause: Database initialization parameters are not set properly. Solution: Set the following initialization parameters: SHARED_POOL_SIZE = 250 Mb or greater LARGE_POOL_SIZE = 150 Mb or greater JAVA_POOL_SIZE = 150 Mb or greater Then rerunning: sqlplus "/ as sysdba" SQL> shutdown immediate SQL> startup upgrade SQL> spool my_upgrade.log SQL> @/rdbms/admin/catpupgrd.sql SQL> @/rdbms/admin/utlrp.sql SQL> spool off Case 3: Shutdown Database Error ORA-600 [Librarycachenotemptyonclose] Symptom: The first shutdown after upgrade to 10g shows an error message: SQL> shutdown immediate Database closed. Database dismounted. ORA-00600: internal error code, arguments: [LibraryCacheNotEmptyOnClose], [], [], [], [], [], [], [] Cause: Related note 435926.1 - Shutdown Database Erroring ORA-600 [Librarycachenotemptyonclose] An unpublished bug: 4483084 say that this particular ORA-600 error can be ignored as it means nothing - actually there is no error. Solution: Do nothing… When it will be available, upgrade to 10.2.0.4.0. Case 4: OLAP Catalog component (AMD) is invalid after upgrade Symptom: Check OLAP catalog state: select comp_name, status, version from DBA_REGISTRY where comp_name= 'OLAP Catalog'; COMP_NAME STATUS VERSION OLAP Catalog INVALID 10.2.0.3.0 Solution: Recreate OLAP Catalog. ORACLE_HOME=/oracle/livedb/10.2 ORACLE_SID=TST10 PATH=$ORACLE_HOME/bin:$PATH sqlplus /nolog SQL>spool olap_spool.log SQL>connect SYS as SYSDBA SQL>@?/olap/admin/olap.sql SYSAUX TEMP; -- Recompile everything SQL>@?/rdbms/admin/utlrp.sql SQL>spool off Check status of AMD again – should be VALID: select comp_name,version,status from dba_registry where comp_name like '%OLAP%'; Case 5: A lot of invalid objects in DB after upgrade Symptom: A lot of invalid objects in DB which cannot be compiled because of this error: (examples) Compilation errors for PACKAGE BODY APPS.HR_DM_LIBRARY #13#10Error: PLS-00907: cannot load library unit PUBLIC.DBMS_SQL (referenced by APPS.HR_DM_LIBRARY) Line: 0 Compilation errors for PACKAGE BODY APPS.HR_DU_DI_INSERT #13#10Error: PLS-00907: cannot load library unit PUBLIC.UTL_FILE (referenced by APPS.HR_DU_DI_INSERT) Line: 0 Compilation errors for PACKAGE BODY APPS.IBE_QUOTE_SAVE_PVT #13#10Error: PLS-00907: cannot load library unit PUBLIC.UTL_HTTP (referenced by APPS.CZ_CF_API) Line: 0 Cause: Existing PL/SQL modules should recompiled in the FORMAT required by the NEW DATABASE. The utlirp.sql hasn't been run after the upgrade. Solution: conn /as sysdba shutdown startup upgrade @$ORACLE_HOME/rdbms/admin/utlirp.sql shutdown startup @$ORACLE_HOME/rdbms/admin/utlrp.sql About the utlirp.sql script: The utlirp.sql script invalidate all existing PL/SQL modules (procedures, functions, packages, types, triggers, views) in a database. This script must be run when it is necessary to regenerate the compiled code because the PL/SQL code format is inconsistent with the Oracle executable. Please note that this script does not recompile invalid objects automatically so you must restart the database and explicitly invoke utlrp.sql to recompile invalid objects. Hope you won't get into this errors, but if so, you can use this post as a reference... :-) Feel free to leave a comment or to share your experience with 10g upgrade! You can also register for email subscription to this blog using the box at the right side menu bar. Aviad

Login Page return "Page Not Found" Error

Today I cloned a new test environment on a new server, after all services startup I tried to get the login page by http://ServerName.domain:port/OA_HTML/AppsLocalLogin.jsp but I got the error message “The page cannot be found”. I tried also http://ServerName.domain:port and I got the same error message. The Apache error_log show: (55.66.77.88 – server ip, 77.88.1.99 – my ip) [Mon Sep 24 10:57:13 2007] [error] OPM: EW: Fail to start process with mod=JServ and grp=DiscoGroup, it's possible that your configuration file is not correct. [Mon Sep 24 10:57:13 2007] [error] OPM: EW: Fail to start process with mod=JServ and grp=OACoreGroup, it's possible that your configuration file is not correct. [Mon Sep 24 10:57:13 2007] [error] OPM: EW: Fail to start process with mod=JServ and grp=XmlSvcsGrp, it's possible that your configuration file is not correct. … … [Mon Sep 24 12:07:00 2007] [error] [client 55.66.77.88] client denied by server configuration: /oracle/testcomn/portal/TEST_servername/oprocmgr-service [Mon Sep 24 12:07:01 2007] [error] [client 55.66.77.88] client denied by server configuration: /oracle/testcomn/portal/TEST_servername/oprocmgr-service [Mon Sep 24 12:07:01 2007] [error] [client 55.66.77.88] client denied by server configuration: /oracle/testcomn/portal/TEST_servername/oprocmgr-service [Mon Sep 24 12:07:01 2007] [error] [client 55.66.77.88] client denied by server configuration: /oracle/testcomn/portal/TEST_servername/oprocmgr-service … … [Mon Sep 24 12:14:03 2007] [error] OPM:Can not find one alive process [Mon Sep 24 12:14:03 2007] [error] [client 77.88.1.99] File does not exist: Client denied by server configuration??!! Why?! I checked all Apache configuration files - httpd.conf, httpds.conf, apps.conf, oprocmgr.conf, httpd_pls.conf - but they all were good… Because it’s a new server I also checked required Linux RPM’s that may be missing, but they all were installed. The content of /etc/hosts file was: 127.0.0.1 localhost.localdomain localhost ServerName 55.66.77.88 ServerName We change it to: 127.0.0.1 localhost.localdomain localhost 55.66.77.88 ServerName.domain ServerName After restart Apache (+ clear cache $COMMON_TOP/_pages/_oa__html/) we got the login screen! :-) Unfortunately, just after resolving this issue I found a related note in Metalink: Note:372096.1 - Servlet and JSP Pages return a "Page Not Found" Error After Install of Applications. Aviad