Resolving Error ORA-28353: Failed to Open Wallet

Full disclosure: this is a post I've had in draft mode for almost one and a half years.

I noticed the original error after applying the October 2018 bundle patch (BP) for 11.2.0.4. While I realize most clients are no longer in 11.2.0.4, this information remains valid for anyone upgrading from 11.2 to 12, 18 or 19c.

I had been doing several tests on my Spanish RAC (Real Application Cluster) Attack for 12.2. The goal was to patch my client to October 2018 PSU; obtaining enough security leverage to avoid patching their database and do their DB (database) upgrade to 18c. I created RAC VMs to enable testing. I also set up my environment to match the client's, which had TDE with FIPS 140 enabled (I will provide more details on this later in the post).

While the patching was successful, the problem arose after applying the patch. I was unable to open the database despite having the correct password for the encryption key.

SQL> alter database open;
 alter database open
 *
 ERROR at line 1:
 ORA-28365: wallet is not open
 
 SQL> alter system set encryption key identified by “xxx”;
 alter system set encryption key identified by “xxxx”
 *
 ERROR at line 1:
 ORA-28353: failed to open wallet
 
 SQL> select * from v$encryption_wallet;
 
 INST_ID WRL_TYPE
 ---------- --------------------
 WRL_PARAMETER
 ------------------------
 STATUS
 ------------------
 1 file
 /etc/oracle/wallets/$ORACLE_UNQNAME
 CLOSED
 
 SQL> ALTER SYSTEM SET ENCRYPTION WALLET OPEN IDENTIFIED BY *****;
 ALTER SYSTEM SET ENCRYPTION WALLET OPEN IDENTIFIED BY *****
 *
 ERROR at line 1:
 ORA-28353: failed to open wallet

When I tried to open the database, this is what appeared in the alert.log:

Tue Feb 16 11:21:56 2018
 QMNC started with pid=81, OS id=86184
 kcbztek_get_tbskey: decrypting encrypted key for tablespace 16 without opening the wallet

I did a rollback of the patch, and as soon as I rolled back the patch, the database opened:

SQL> select name,open_mode ,database_role from v$database;
 
 NAME OPEN_MODE DATABASE_ROLE
 --------- -------------------- ----------------
 TEST READ WRITE PRIMARY

After many days of looking for information to address the error, I noticed that FIPS 140-2 was enabled. FIPS (Federal Information Processing Standard), 140-2, is a US government standard defining cryptographic module security requirements.

You can see it's enabled for SSL in the following file:

[oracle@collabn1 /u01/app/oracle/product/11.2.0.4/dbhome_1/ldap/admin ]$ cat fips.ora
 #
 # file to be reside in $ORACLE_HOME/ldap/admin
 #
 # sec-220
 #
 sslfips_140=true

I was able to find a document called After Applying October 2018 CPU/PSU, Auto-Login Wallet Stops Working For TDE With FIPS Mode Enabled (Doc ID 2474806.1).

This helped me discover the solution is to patch the DB with October 2018 PSU and, after patching the binaries, recreate the auto login file cwallet.sso with a compatibility of version 12. After you have done this, you will be able to open your DB normally.

[oracle@collabn1 ~]$ cd /etc/oracle/wallets/test
 [oracle@collabn1 /etc/oracle/wallets/test]$ mv cwallet.sso cwallet.sso.original
 
 [oracle@collabn1 /etc/oracle/wallets/test]$ orapki wallet create -wallet /etc/oracle/wallets/test -auto_login -compat_v12
 Oracle PKI Tool : Version 11.2.0.4.0 - Production
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
 Enter wallet password: ******
 
 [oracle@collabn1 ~]$ srvctl status database -d test
 Instance test1 is running on node collabn1
 Instance test2 is running on node collabn2
 
 SQL> select name,open_mode ,database_role from v$database;
 
 NAME OPEN_MODE DATABASE_ROLE
 --------- -------------------- ----------------
 TEST READ WRITE PRIMARY
 
 SQL> select * from v$encryption_wallet;
 
 INST_ID WRL_TYPE
 ---------- --------------------
 WRL_PARAMETER
 ------------------------
 STATUS
 ------------------
 1 file
 /etc/oracle/wallets/$ORACLE_UNQNAME
 OPEN
 

IMPORTANT: DO NOT recreate the ewallet.p12 file! This will likely cause data loss, as you will lose the master key required to decrypt your encrypted data.

Below is an example of what you DO NOT WANT TO DO:

[oracle@collabn1 ~]$ cd /etc/oracle/wallets/test
 [oracle@collabn1 /etc/oracle/wallets/test]$ mv ewallet.p12 ewallet.p12.old
 
 [oracle@collabn1 /etc/oracle/wallets/test]$ sqlplus / as sysdba
 SQL> ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY Welcome1 WITH BACKUP;
 
 keystore altered
 

It's important to note that the above also applies to Jan 2019 Database BP, or to any upgrade from 11.2.0.4 to 12, 18 or 19c. This means you will face this issue for anything after October 2018 if you are using TDE and SSL with FIPS.

Note: This was originally posted in rene-ace.com.