How to Migrate Oracle Applications 11.5.10.2 from on-premises SunOS fleet to Linux on AWS

 

 

It’s no secret that many companies have been moving existing on-premises environments to the cloud. They all have different reasons and goals:

1. Get rid of the old hardware fleet by moving Oracle Applications over to a modern cloud solution.
2. Combine a move to the cloud with an Oracle Applications upgrade project.
3. Use Sunset Oracle applications for archiving purposes.

In this post, I’d like to to share some lessons learned while working on option #3 from the above list.

 

 

Background

My goal was to move existing Oracle Applications 11.5.10.2 from on-premises Sun Solaris fleet to Linux VMs running on Amazon Web Services (AWS). Source and destination configuration were also to be preserved. Details regarding the source and destination environments are as follows:

• Source
  • Oracle Applications 11.5.10.2 running on Solaris SPARC 5.10.
  • Multi-node implementation dedicated to database node running concurrent processing (admin) node on the same host.
  • Web / Forms services running on a separate host.
• Target
  • No version change, Oracle Applications 11.5.10.2 running on Oracle Linux 5.11.
  • Configuration remains the same: database node and concurrent manager processing (admin) node on one host.
  • Web / Forms services running on a separate host.

It’s important to remember that Oracle Applications 11.5.10.2 is certified in split configuration model. According to the split configuration, a model database tier can be 32-bit or 64-bit but the application tier is only 32-bit. As per the certification grid, Linux 5 is the latest version certified. As a result, Oracle Linux 5.11 was chosen as the target platform. Both target hosts are 64-bit in split configuration, emulating 32-bit environments for application processes.

Reference : Oracle Applications Installation Update Notes, Release 11i (11.5.10.2) (Doc ID 316806.1) 

Let’s review the possible issues that might come up while migrating older technology components – RDBMS: 10.2.0.3 / Oracle Applications: 11.5.10.2 / iAS 1.0.2.2.2 rollup 5 / Forms 6.0 (Forms Runtime) Version 6.0.8.28.0.

Case 1

Where to find older versions of Oracle Linux in AWS?

Solution 1

Please refer to the following notes: Launch an Oracle Linux Instance in AWS.

The link covers all necessary details regarding available Oracle Linux versions and how to find them. We used: AMI ID: ami-c28e21b5, AMI Name: OL5.11-x86_64-10-17-2014-ebs

Case 2

The import process dies with the error message LPX-00225. During the full database import process, you get a fatal error and the whole import process is terminated. The initial approach would be to exclude failing triggers and re-export those manually, however, that method is time-consuming:

ORA-39125: Worker unexpected fatal error in KUPW$WORKER.PUT_DDLS while calling DBMS_METADATA.CONVERT [TRIGGER:"APPS"."AX_AP_INVOICES_ARU1"]
ORA-06502: PL/SQL: numeric or value error
LPX-00225: end-element tag "BODY" does not match start-element tag "WHENCLAUSE"
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: at "SYS.KUPW$WORKER", line 6234
----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
0xb0a4c4f0     14916  package body SYS.KUPW$WORKER
0xb0a4c4f0      6293  package body SYS.KUPW$WORKER
0xb0a4c4f0     12280  package body SYS.KUPW$WORKER
0xb0a4c4f0      3278  package body SYS.KUPW$WORKER
0xb0a4c4f0      6882  package body SYS.KUPW$WORKER
0xb0a4c4f0      1259  package body SYS.KUPW$WORKER
0x8a0fe738         2  anonymous block

Solution 2

Reason Oracle RDBMS 10.2.0.3 and other versions DataPump export/import might fail to import with ORA-39125 errors is if the trigger text on the source database has trailing null characters in the WHEN clause.

As a workaround, apply patch 4352110 to source and destination database homes. In addition, on source apply, patch special post steps to reload packages. Rerun export and import data to the destination:

SQL> @?/rdbms/admin/catdph.sql
SQL> @?/rdbms/admin/catdpb.sql

It’s possible to get an ORA-7445 error while running above cat* scripts:

ORA-07445: exception encountered: core dump [strlen()+24] [SIGSEGV] [Address not mapped to object]

As a workaround, disable the following init parameters:

*.audit_sys_operations
*.audit_trail
*.audit_syslog_level

References:

DataPump Import (IMPDP) Of Triggers Fails With Errors ORA-39126 ORA-6502 LPX-225 ORA-6512 (Doc ID 970962.1)

Bug 4352110 – ORA-39125 from expdp/impdp of triggers with nulls in WHEN clause (Doc ID 4352110.8)

Patch 4352110: IMPORT EXITS WITH ORA-39125 UNEXPECTED FATAL ERROR IN KUPW$WORKER.PUT_DDLS

ORA-07445: exception encountered: core dump [strlen()+24][SIGSEGV] while executing utlu(nnn)i.sql or when DBUA executing preupgrade script (Doc ID 1520710.1)

Case 3

If the Oracle Application system is out of support or frozen during a longer period of time, however, the OS is being kept updated, $AD_TOP/bin/admkappsutil.pl command might complete with error.

# perl $AD_TOP/bin/admkappsutil.pl
Starting the generation of appsutil.zip
Log file located at /......../MakeAppsUtil_10000000.log
output located at /......../appsutil.zip
MakeAppsUtil completed with errors, Please check logfile for more information.

Solution 3

The application file system doesn’t have the necessary files to support newer versions of zip and unzip utilities. Verify versions of zip and unzip utilities. Apply patch 9874305 to the source application system and redeploy the application tier file system to the destination.

Reference: admkappsutil.pl Fails With Error ‘MakeAppsUtil completed with errors, Please check logfile for more information.’ (Doc ID 1584104.1)

Case 4

You might receive multiple re-link errors while migrating the platform patch applications. Example relink error:

/usr/bin/ld: /......../ad/11.5.0/lib/adjkey.o: Relocations in generic ELF (EM: 2)
/usr/bin/ld: /......../ad/11.5.0/lib/adjkey.o: Relocations in generic ELF (EM: 2)
/......../ad/11.5.0/lib/adjkey.o: could not read symbols: File in wrong format
collect2: ld returned 1 exit status
make: *** [/......../appl/ad/11.5.0/bin/adjkey] Error 1
Done with link of ad executable 'adjkey' on Tue Nov  2 09:39:38 EDT 2021

Solution 4

Before proceeding with manifest file upload to My Oracle Support, please ensure that an updated snapshot of the information is provided. Not doing so may result in confusion and useless troubleshooting of re-linking issues. Maintain snapshot information using AdAdmin and recreate the manifest file. There might  be confusion finding the URL for requesting the platform migration patch. Use URL: PlatformMigration

References:

Migrating to Linux with Oracle Applications Release 11i (Doc ID 238276.1)

Relink Error During Manifest Patch Application During Platform Migration Process (Doc ID 1307460.1)

Case 5

AutoConfig run hangs on the target application tier file system when calling adunzip.pl.

Solution 5

Verify location $COMMON_TOP/util/unzip/unzip. In case location is empty 5.50 UNZIP version not present under $COMMON_TOP. As next step, verify $COMMON_TOP/util/unzip/<your_platform> for file unz550x-glibc.tar.gz. Copy file unz550x-glibc.tar.gz to $COMMON_TOP/util/unzip/unzip. Unarchive file.

Another location where unz550x-glibc.zip can be found is $AD_TOP/resource/3rdparty/stdalone. The same instructions apply.

Re-launch AutoConfig. Repeat the same steps on other application tier hosts if needed.

Case 6

This is reasonable to process sanity checks on the target environment. Verify front-end access, test login, open forms. Part of the sanity test is to verify the concurrent manager subsystem.  When scheduling a new concurrent program to run on target, there’s a chance for an APP-FND-00178 error. You get it when submitting a new concurrent program to run.

Solution 6

The reason for the issue is that export on source was performed while source instance was running. Max ID values for FND_LOGINS and FND_CONCURRENT_REQUESTS tables are higher than sequence values:

0. Stop concurrent manager subsystem

1. Review fnd_logins ID and corresponding sequence status

SQL> select max(LOGIN_ID) from fnd_logins;

SQL> select fnd_logins_s.nextval from sys.dual;

2.  Perform the same check on fnd_concurrent_requests table as well using the below queries:

SQL> select max(request_id) from fnd_concurrent_requests;

SQL> select fnd_concurrent_requests_s.nextval from dual;

3. Either recreate sequence with higher START WITH value or roll up sequence by selecting nextval from dual

4. Start concurrent manager subsystem and test by launching new concurrent program

Case 7

You might need to recreate the CTXSYS schema or recreate the Text index. You might receive different errors while working with text indexes on the destination environment. There are a number of solutions to fix this.

Solution 7

It’s not unusual if  text index user preferences aren’t included in the export dump. After the import, Text indexes are usable. Since the user-defined preferences are not exported/imported, errors are generated when rebuilding or creating new text indexes using these preferences.

References:

Why after Export/Import All User Text Index Preferences Are Lost ? (Doc ID 468599.1)

How to Use CTX_REPORT to Get Information on Your Text Indexes (Doc ID 189819.1)

Following My Oracle Support notes provides a complete list of seeded Text indexes used in Oracle Application 11.5.10.2. They also provide scripts as well as necessary parameters to recreate indexes and preferences.

Reference: Oracle Text: Re-installation of Applications 11i (11.5.10) Oracle Text Indexes (Doc ID 312640.1)

Use below SQL statements to compare Text index count and index preference count in source and target.

select idx_owner, idx_name from ctxsys.ctx_indexes order by 1,2;

select * from ctxsys.ctx_preferences order by 1,2;

You might encounter a deprecated KOREAN_LEXER. KOREAN_LEXER was deprecated in RDBMS version 10.2.0.3 and replaced by KOREAN_MORPH_LEXER. Few index preferences use KOREAN_MORPH_LEXER. To recreate particular Text indexes and their preferences, apply patch 5079874 on the source and target environments.

Reference: cssrctxp.sql fails with DRG-10703: Invalid Framework Object Korean_lexer (Doc ID 402422.1)

AMV Text index recreation will also fail, so use the fix provided below to correct this:

(1) backup file  $AMV_TOP/patch/115/sql/amviimt.sql

(2) Change line 135 in $AMV_TOP/patch/115/sql/amviimt.sql

FROM:
ctx_ddl.create_preference('korean_lexer',   'KOREAN_LEXER');

TO:
ctx_ddl.create_preference('korean_lexer',   'KOREAN_MORPH_LEXER'); 

AutoConfig run might also fail while processing $JTF_TOP:

DECLARE
*
ERROR at line 1:
ORA-29874: warning in the execution of ODCIINDEXALTER routine
ORA-29960: line 1,
DRG-10595: ALTER INDEX JTF_AMV_ITEMS_URL_CTX failed
DRG-10561: index JTF_AMV_ITEMS_URL_CTX is not valid for requested operation
ORA-06512: at "SYS.DBMS_SYS_SQL", line 909
ORA-06512: at "SYS.DBMS_SQL", line 39
ORA-06512: at line 96

As discussed above, recreate corresponding Text indexes under JTF owner:

Reference: Running AutoConfig on a Cloned Application Tier Fails In Script jtfictx.sh With Errors ORA-29874, DRG-10595 (Doc ID 1271186.1)

A few points to review before migrating

I suggest reviewing the following My Oracle Support notes to ensure you’re well prepared.

To avoid possible database export/import, review the Checklist For Slow Performance Of DataPump Export (expdp) And Import (impdp) (Doc ID 453895.1). You can find some useful patches to be applied on target as well as on destination. Another option to increase export/import performance is to exclude statistics from export or apply to exclude during import. If there are such schemas that aren’t needed on target, exclude those. For example, schema containing PERFSTAT can be excluded and re-installed later.

To avoid confusion when receiving an FRM-40031 error message when verifying forms on target environment, use AdAdmin and recompile forms, libraries and menu files.

Check out these useful links for  moving an EBS environment to Linux:

Requirements For Installing Oracle10gR2 On RHEL 5/OL 5 (x86_64) (Doc ID 421308.1)

Migrating to Linux with Oracle Applications Release 11i (Doc ID 238276.1)

Upgrading Developer 6i with Oracle Applications 11i (Doc ID 125767.1)

 

I hope you find this post useful. Feel free to leave questions in the comments and sign up for the next post here.