How to Troubleshoot OEM 12c Cloud Control Auto-Discovery

I was recently involved in an upgrade project to go from 11.2.0.2 to 11.2.0.3 on an Exadata V2. We hit some snags during the upgrade specifically related to OEM 12c Cloud Control. We performed an out-of-place upgrade and OEM 12.1.0.1.0 had some difficulty in dealing with this.

12c Cloud Control is supposed to run a daily check that looks for new targets on each server. When it finds something new, it places this in a queue to wait for admin approval. With a single click, you can promote the newly discovered target to an OEM managed object.

• You can manually trigger this process. Go to Setup -> Add Target -> Configure Auto-Discovery -> Multiple Target-Type Discovery: Configure/Wrench-Icon -> Run Discovery Now
• You can view the queue and promote targets by navigating to Setup -> Add Target -> Auto-Discovery Results

It looks like a very nice process – except that it’s not working at all on our system. The stuff currently in our queue hasn’t existed on the server for years, and the new stuff is not getting detected and added to the queue. I haven’t completely solved this yet, but I have learned a few things in the process of working on it.

A Few Useful References

First off, a few useful references about auto-discovery. OEM is very extensible, and you can create your own target types and write your own auto-discovery scripts.

• Good overview with lots of pictures: https://docs.oracle.com/cd/E24628_01/doc.121/e24473/discovery.htm
• Details about how discovery scripts are implemented and how they work: https://docs.oracle.com/cd/E24628_01/doc.121/e25161/discover.htm

Key takeaway: Auto-discovery is driven by a script that is sitting on the server, generally written in Perl. This script does not change anything when it runs – it only generates output: a list of all targets that it currently sees. OEM sorts out which are new. This means that it’s safe to manually run the auto-discovery script on the server and see if it’s working as expected.

Additional note: Some scripts will also write debug information (info/debug/trace) to emagent_perl.trc in the Management Agent log directory.

Manually Running Auto-Discovery

It took me a few tries to get all the library references right, but you can now just copy and paste the commands below to run the Oracle Home auto-discovery yourself. There should only be minor tweaks to run other auto-discovery scripts.


[oracle@server:~]$ cd $AGENT_HOME
[oracle@server:/u01/app/oracle/product/12.1]$ export LD_LIBRARY_PATH=$AGENT_HOME/core/12.1.0.1.0/lib
[oracle@server:/u01/app/oracle/product/12.1]$ core/12.1.0.1.0/perl/bin/perl -Icore/12.1.0.1.0/sysman/admin/scripts -Icore/12.1.0.1.0/perl/lib/site_perl/5.10.0/x86_64-linux-thread-multi plugins/oracle.sysman.oh.discovery.plugin_12.1.0.1.0/OracleHomeDiscovery.pl

<Targets>
<Target TYPE=”oracle_home” NAME=”Ora11g_gridinfrahome1_1_server” DISPLAY_NAME=”Ora11g_gridinfrahome1_server”>
<Property NAME=”HOME_TYPE” VALUE=”O” />
<Property NAME=”INSTALL_LOCATION” VALUE=”/u01/app/11.2.0.3/grid” />
<Property NAME=”INVENTORY” VALUE=”/u01/app/oraInventory” />
</Target>
<Target TYPE=”oracle_home” NAME=”Ora11g_gridinfrahome2_4_server” DISPLAY_NAME=”Ora11g_gridinfrahome2_server”>
<Property NAME=”HOME_TYPE” VALUE=”O” />
<Property NAME=”INSTALL_LOCATION” VALUE=”/u01/app/11.2.0.2/grid” />
<Property NAME=”INVENTORY” VALUE=”/u01/app/oraInventory” />
</Target>
<Target TYPE=”oracle_home” NAME=”OraDb11g_home1_2_server” DISPLAY_NAME=”OraDb11g_home1_server”>
<Property NAME=”HOME_TYPE” VALUE=”O” />
<Property NAME=”INSTALL_LOCATION” VALUE=”/u01/app/oracle/product/11.2.0.3/dbhome_1″ />
<Property NAME=”INVENTORY” VALUE=”/u01/app/oraInventory” />
</Target>
<Target TYPE=”oracle_home” NAME=”OraDb11g_home2_5_server” DISPLAY_NAME=”OraDb11g_home2_server”>
<Property NAME=”HOME_TYPE” VALUE=”O” />
<Property NAME=”INSTALL_LOCATION” VALUE=”/u01/app/oracle/product/11.2.0.2/dbhome_1″ />
<Property NAME=”INVENTORY” VALUE=”/u01/app/oraInventory” />
</Target>
<Target TYPE=”oracle_home” NAME=”agent12c1_6_server” DISPLAY_NAME=”agent12c1_server”>
<Property NAME=”HOME_TYPE” VALUE=”O” />
<Property NAME=”INSTALL_LOCATION” VALUE=”/u01/app/oracle/product/12.1/core/12.1.0.1.0″ />
<Property NAME=”INVENTORY” VALUE=”/u01/app/oraInventory” />
</Target>
<Target TYPE=”oracle_home” NAME=”sbin12c1_7_server” DISPLAY_NAME=”server”>
<Property NAME=”HOME_TYPE” VALUE=”O” />
<Property NAME=”INSTALL_LOCATION” VALUE=”/u01/app/oracle/product/12.1/sbin” />
<Property NAME=”INVENTORY” VALUE=”/u01/app/oraInventory” />
</Target>
</Targets>

Update Feb 18: You can get all four nodes at once by copying the following dcli command:

dcli -l oracle -g ~/dbs_group LD_LIBRARY_PATH=$AGENT_HOME/core/12.1.0.1.0/lib $AGENT_HOME/core/12.1.0.1.0/perl/bin/perl -I $AGENT_HOME/core/12.1.0.1.0/sysman/admin/scripts -I $AGENT_HOME/core/12.1.0.1.0/perl/lib/site_perl/5.10.0/x86_64-linux-thread-multi $AGENT_HOME/plugins/oracle.sysman.oh.discovery.plugin_12.1.0.1.0/OracleHomeDiscovery.pl

Manually Adding Targets

After you’ve run that script, you can manually add the targets using the exact outputs from the auto-discovery script. In most situations, you probably don’t need to do this, but it’s neat that there’s a one-to-one correlation between the manual add parameters and the script output.

Manually Addding Oracle Home

I’m curious – has anyone else ever had to troubleshoot auto-discovery? Care to share about your experience? Any additional tips you can add?