Pythian Blog: Technical Track

Troubleshooting a GoldenGate error code

After applying Oracle GoldenGate V12.2.0.1.170919 for Oracle Database 12c OPTIMIZER Patch for Bug# 26849949, starting GoldenGate extract failed with OGG-00303: Unable to open credential store. Error code 43,490. Here is what the report looks like.
$ head -50 E_LAX6.rpt
 ***********************************************************************
  Oracle GoldenGate Capture for Oracle
  Version 12.2.0.1.170919 OGGCORE_12.2.0.1.0OGGBP_PLATFORMS_171030.0908_FBO
  Linux, x64, 64bit (optimized), Oracle 12c on Oct 30 2017 20:59:41
  
 Copyright (C) 1995, 2017, Oracle and/or its affiliates. All rights reserved.
 
 
  Starting at 2018-10-25 15:08:33
 ***********************************************************************
 
 Operating System Version:
 Linux
 Version #2 SMP Wed Jul 11 12:11:36 PDT 2018, Release 4.1.12-94.8.5.el7uek.x86_64
 Node: localhost
 Machine: x86_64
  soft limit hard limit
 Address Space Size : unlimited unlimited
 Heap Size : unlimited unlimited
 File Size : unlimited unlimited
 CPU Time : unlimited unlimited
 
 Process id: 23154
 
 Description: 
 
 ***********************************************************************
 ** Running with the following parameters **
 ***********************************************************************
 
 2018-10-25 15:08:33 INFO OGG-03059 Operating system character set identified as UTF-8.
 
 2018-10-25 15:08:33 INFO OGG-02695 ANSI SQL parameter syntax is used for parameter parsing.
 EXTRACT e_lax
 USERIDALIAS gguser
 
 Source Context :
  SourceModule : [er.init]
  SourceID : [/scratch/aime/adestore/views/aime_adc4150431/oggcore/OpenSys/src/app/er/init.cpp]
  SourceFunction : [get_infile_params]
  SourceLine : [5554]
  ThreadBacktrace : [11] elements
  : [/u01/gg/12.2.0/libgglog.so(CMessageContext::AddThreadContext()+0x1b) [0x7f714024709b]]
  : [/u01/gg/12.2.0/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x135) [0x7f7140241165]]
  : [/u01/gg/12.2.0/libgglog.so(_MSG_ERR_STARTUP_PARAMERROR_ERRORTEXT(CSourceContext*, char const*, CMessageFactory::MessageDisposition)+0x30) [0x7f71402308d0]]
  : [/u01/gg/12.2.0/extract(get_infile_params(time_elt_def*, time_elt_def*, char**, ggs::gglib::ggdatasource::DataSourceParams&, ggs::Heartbeat::MapGeneratorParams&)+0x5da1) [0x5c4c91]]
  : [/u01/gg/12.2.0/extract() [0x5f036a]]
  : [/u01/gg/12.2.0/extract(ggs::gglib::MultiThreading::MainThread::ExecMain()+0x60) [0x6cea60]]
  : [/u01/gg/12.2.0/extract(ggs::gglib::MultiThreading::Thread::RunThread(ggs::gglib::MultiThreading::Thread::ThreadArgs*)+0x14d) [0x6cfcdd]]
 
  Since I did not have the password for database gguser, I modified the password from the database and recreated credentialstore using ggsci.
  
 info credentialstore
 delete credentialstore
 create wallet.
 add credentialstore
 alter credentialstore add user gguser alias gguser.
 alter credentialstore add user GGUSER alias GGUSER
 info credentialstore
 
The reason we check credentialstore before deleting is to determine aliases.
GGSCI 1> info credentialstore
 
 Reading from ./dircrd/:
 
 Default domain: OracleGoldenGate
 
  Alias: gguser
  Userid: gguser
 
  Alias: GGUSER
  Userid: GGUSER
 
Here are the steps from ggserr.log.
2018-10-25 15:44:00 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (ggsuser): create wallet.
 2018-10-25 15:44:00 INFO OGG-02096 Oracle GoldenGate Command Interpreter for Oracle: Created wallet at location 'dirwlt'.
 2018-10-25 15:44:00 INFO OGG-02096 Oracle GoldenGate Command Interpreter for Oracle: Opened wallet at location 'dirwlt'.
 2018-10-25 15:44:10 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (ggsuser): add credentialstore.
 2018-10-25 15:44:10 INFO OGG-02096 Oracle GoldenGate Command Interpreter for Oracle: Credential store created in ./dircrd/.
 2018-10-25 15:44:22 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (ggsuser): alter credentialstore add user gguser alias gguser.
 2018-10-25 15:44:31 INFO OGG-02096 Oracle GoldenGate Command Interpreter for Oracle: Credential store in ./dircrd/ altered.
 2018-10-25 15:44:55 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (ggsuser): alter credentialstore add user GGUSER alias GGUSER.
 2018-10-25 15:45:06 INFO OGG-02096 Oracle GoldenGate Command Interpreter for Oracle: Credential store in ./dircrd/ altered.
 2018-10-25 15:45:15 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (ggsuser): info credentialstore.
 2018-10-25 15:45:15 INFO OGG-02096 Oracle GoldenGate Command Interpreter for Oracle: Reading from ./dircrd/:.
 
Here are the permissions for the files.
$ chmod 775 -R dirwlt/ dircrd/
 $ ls -l dirwlt/ dircrd/
 dircrd/:
 total 4
 -rwxrwxr-x 1 gguser oinstall 701 Oct 25 15:45 cwallet.sso
 
 dirwlt/:
 total 4
 -rwxrwxr-x 1 gguser oinstall 290 Oct 25 15:44 cwallet.sso
 
Starting extract failed again!
2018-10-25 15:45:30 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (gguser): dblogin useridalias gguser.
 2018-10-25 15:45:35 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (gguser): start e*.
 2018-10-25 15:45:35 INFO OGG-00963 Oracle GoldenGate Manager for Oracle, mgr.prm: Command received from GGSCI on host 10.80.27.191:39060 (START EXTRACT E_LAX ).
 2018-10-25 15:45:35 INFO OGG-00960 Oracle GoldenGate Manager for Oracle, mgr.prm: Access granted (rule #5).
 2018-10-25 15:45:35 INFO OGG-00975 Oracle GoldenGate Manager for Oracle, mgr.prm: EXTRACT E_LAX starting.
 2018-10-25 15:45:35 INFO OGG-00992 Oracle GoldenGate Capture for Oracle, e_lax.prm: EXTRACT E_LAX starting.
 2018-10-25 15:45:35 INFO OGG-03059 Oracle GoldenGate Capture for Oracle, e_lax.prm: Operating system character set identified as UTF-8.
 2018-10-25 15:45:35 INFO OGG-02695 Oracle GoldenGate Capture for Oracle, e_lax.prm: ANSI SQL parameter syntax is used for parameter parsing.
 2018-10-25 15:45:35 ERROR OGG-00303 Oracle GoldenGate Capture for Oracle, e_lax.prm: Unable to open credential store. Error code 43,490.
 2018-10-25 15:45:35 ERROR OGG-01668 Oracle GoldenGate Capture for Oracle, e_lax.prm: PROCESS ABENDING.
 2018-10-25 15:45:38 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (gguser): info all.
 
Set environment variables for ORACLE_HOME and ORACLE_SID for extract, and restart extract solved the issue.
2018-10-25 15:52:44 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (ggsuser): start e*.
 2018-10-25 15:52:44 INFO OGG-00963 Oracle GoldenGate Manager for Oracle, mgr.prm: Command received from GGSCI on host 10.80.27.191:39169 (START EXTRACT E_LAX ).
 2018-10-25 15:52:44 INFO OGG-00960 Oracle GoldenGate Manager for Oracle, mgr.prm: Access granted (rule #5).
 2018-10-25 15:52:44 INFO OGG-00975 Oracle GoldenGate Manager for Oracle, mgr.prm: EXTRACT E_LAX starting.
 2018-10-25 15:52:44 INFO OGG-00992 Oracle GoldenGate Capture for Oracle, e_lax.prm: EXTRACT E_LAX starting.
 2018-10-25 15:52:44 INFO OGG-03059 Oracle GoldenGate Capture for Oracle, e_lax.prm: Operating system character set identified as UTF-8.
 2018-10-25 15:52:44 INFO OGG-02695 Oracle GoldenGate Capture for Oracle, e_lax.prm: ANSI SQL parameter syntax is used for parameter parsing.
 2018-10-25 15:52:44 INFO OGG-02095 Oracle GoldenGate Capture for Oracle, e_lax.prm: Successfully set environment variable ORACLE_HOME=/u01/app/oracle/product/12.1.0/db_2.
 2018-10-25 15:52:44 INFO OGG-02095 Oracle GoldenGate Capture for Oracle, e_lax.prm: Successfully set environment variable ORACLE_SID=sourcedb.
 2018-10-25 15:52:44 INFO OGG-02095 Oracle GoldenGate Capture for Oracle, e_lax.prm: Successfully set environment variable ORACLE_HOME=/u01/app/oracle/product/12.1.0/db_2.
 2018-10-25 15:52:44 INFO OGG-02095 Oracle GoldenGate Capture for Oracle, e_lax.prm: Successfully set environment variable ORACLE_SID=sourcedb.
 2018-10-25 15:52:56 INFO OGG-00993 Oracle GoldenGate Capture for Oracle, e_lax.prm: EXTRACT E_LAX started.
 
Currently, I don't know if patching caused the issue or if it is a pre-existing condition. What's interesting is the same issue happened in a different system where environment variables for ORACLE_HOME and ORACLE_SID were not set for extract. Instead of restarting just the extract, all the processes were stopped and restarted. In summary, it would be a good idea to stop and start Goldengate processes before patching in order to determine any pre-existing conditions.

No Comments Yet

Let us know what you think

Subscribe by email