Oracle Exadata ASR General Troubleshooting

Oracle Auto Service Request (ASR) is a secure, scalable, customer-installable software feature of warranty and Oracle Support Services that provides auto-case generation when common hardware component faults occur. I thought I’d share how we monitor the ASR manager process and how we resolve some ASR issues to keep things running smoothly.

The shell scripts provided by the Oracle Auto Service Request (ASR) manager is a great tool to check the ASR status: /opt/asrmanager/util/check_asr_status.sh. Here’s an example of a good ASR status:

[root@ASRServer ~]# /opt/asrmanager/util/check_asr_status.sh
Checking ASR Manager status ..
PASS: ASR Manager (pid 18350) is RUNNING.
PASS: ASR Manager bundles state is active.
PASS: ASR Manager SNMP listener is running (SNMP port 162).
PASS: ASR Manager HTTP receiver is running (HTTP port 8234).
PASS: ASR Manager version 20.1.0.
PASS: ASR Manager database connectivity is working.
PASS: ASR Manager Registration SSO user name is set correctly.
PASS: ASR Manager Oracle transport connectivity is working.
PASS: ASR Manager Oracle transport endpoint is set correctly. https://transport.oracle.com
PASS: ASR Manager site id xxxxxxxxxxx.
PASS: ASR Manager registration id xxxxxxxxxxx.
PASS: ASR Manager logging level info.
PASS: ASR Manager process is running as root.
PASS: ASR Autoupdate bundles state is active.

You’ll see the ASR failure:

1. FAIL: ASR Manager Oracle transport connectivity is not working.

Solution: for this failure, restarting ASR manager will fix the issue:

[root@ASRServer ~]# /opt/asrmanager/bin/asr restart

2. FAIL: ASR Manager missing registration id.

Solution: For this failure, sometimes restarting ASR Manager may not fix the issue. If I test the connection to the transport server (https://transport.oracle.com/), I see the following error:

[root@ASRServer ~]# cd /opt/asrmanager/bin/
[root@ASRServer /opt/asrmanager/bin]# ./asr test_connection
Connecting to endpoint @ https://transport.oracle.com
Sending ping message.

ASR Manager is enabled.
Transport test failed!

I tried to register transport server but failed:

[root@ASRServer /opt/asrmanager/bin]# ./asr register

1) transport.oracle.com
Select destination transport server or enter full URL for alternate server [1]:


If a proxy server is required for HTTPS communication to the internet,
enter the information below. If no proxy is needed, enter -
Proxy server name []:

An Oracle Single Sign On (OSSO) account is required for data submission.
If you do not have an account or have forgotten your username or
password, visit http://support.oracle.com

Username [xxxxxx@pythian.com]:
Password:
Password again (to verify):

Contacting transport servers. Please wait...

Checking connection to https://transport.oracle.com/v1/
Error: Connection reset


Registration failed.

Tried to do the heartbeat and it failed:

[root@ASRServer /opt/asrmanager/bin]# ./asr heartbeat
Failed to generate heartbeat for ASR Manager due to transport problem.

So far, the test results tell us the issue is related to the network. After checking with the SA team and confirming this, I was able to register the transport server successfully:

[root@ASRServer ~]# /opt/asrmanager/bin/asr register

1) transport.oracle.com
Select destination transport server or enter full URL for alternate server [1]:


If a proxy server is required for HTTPS communication to the internet,
enter the information below. If no proxy is needed, enter -
Proxy server name []:

An Oracle Single Sign On (OSSO) account is required for data submission.
If you do not have an account or have forgotten your username or
password, visit http://support.oracle.com

Username []: xxxxxx@pythian.com
Password:
Password again (to verify):

Contacting transport servers. Please wait...

Checking connection to https://transport.oracle.com/v1/
Connection is ok. Trying to register client.
Successfully registered with Oracle ASR Infrastructure.


Registration complete.
Successfully submitted ASR Registration Request.


Stopped Oracle ASR TimerTask Scheduler
Started Oracle ASR TimerTask Scheduler

3. FAIL: ASR Autoupdate bundles state is NOT active.

Solution: sometimes restarting the ASR manager doesn’t fix the issue. Because the process AUTOUPDATE was stopped, you need to start it up as follows:

[root@ASRServer ~]# cd /opt/asrmanager/bin/
[root@ASRServer /opt/asrmanager/bin]# ./asrautoupdate status
AUTOUPDATE is NOT RUNNING.
[root@ASRServer /opt/asrmanager/bin]# ./asrautoupdate start
[root@ASRServer /opt/asrmanager/bin]# ./asrautoupdate status
AUTOUPDATE (pid 27794) is RUNNING.

4. Oracle ASR: Warning – Connectivity Failure – Assets in “Active – No Heartbeat” state.

Solution: when you get this alert, log into your MOS and check the status of the Exadata assets and you’ll see it’s inactive like the below example:

Now you only need to restart ASR manager service:

[root@ASRServer ~]# /opt/asrmanager/bin/asr restart

Then, you validate the status in MOS:

5. ERROR: ASR Manager(s) on xx.xx.xx.xx are not accessible or do not support automatic diagnostic package upload. Diagnostic packages will not be automatically uploaded for Service Requests.

If you see this error, validate SNMP and you’ll see the next error:

CellCLI> alter cell validate snmp type=asr
ASR Manager(s) on xx.xx.xx.xx are not accessible or do not support automatic diagnostic package upload. Diagnostic packages will not be automatically uploaded for Service Requests.
Cell Exacel01 successfully altered

Here is the attribute of snmpSubscriber:

CellCLI> list cell attributes snmpSubscriber
((host=xx.xx.xx.xx,port=162,community=public,type=ASR,asrmPort=16161))

Solution: The asrm port has been changed from 16161 to 8234. You only need to update the port and then validate:

CellCLI> ALTER cell snmpSubscriber=(host=‘xx.xx.xx.xx’,port=162,community=public,type=ASR,asrmPort=8234))
CellCLI> alter cell validate snmp type=asr

I hope you enjoyed this post. Feel free to ask me questions in the comments.

If you haven’t already, sign up for my next post here.