WebLogic Admin Console 12.2.1.4.0 Is Not Starting Due to Coherence GE 12.2.1.4.0

In this post I’ll address the situation where, after installing Fusion Middleware (FMW) 12.2.0.4, the WebLogic Admin Server won’t start due to an issue reported on Coherence, one of the embedded components of FMW (Forms/Service Bus/SOA Architecture).

You will face this error even if you aren’t using Coherence when starting WebLogic Admin Console.

I recently faced this: Delaying formation of a new cluster; multicast networking appears to be inoperable on interface 192.168.0.100 as this process isn’t receiving even its own transmissions; consider forcing IPv4 via -Djava.net.preferIPv4Stack=true>

<Feb 17, 2021 9:54:04,361 PM UTC> <Warning> <Coherence> <BEA-000000> <2021-02-17 21:54:04.361/355.903 Oracle Coherence GE 12.2.1.4.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; multicast networking appears to be inoperable on interface 192.168.0.100 as this process isn't receiving even its own transmissions; consider forcing IPv4 via -Djava.net.preferIPv4Stack=true>
<Feb 17, 2021 9:54:08,285 PM UTC> <Error> <Coherence> <BEA-000000> <2021-02-17 21:54:08.285/359.828 Oracle Coherence GE 12.2.1.4.0 <Error> (thread=[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
  MemberSet=MasterMemberSet(
    ThisMember=null
    OldestMember=null
    ActualMemberSet=MemberSet(Size=0
      )
    MemberId|ServiceJoined|MemberState|Version
    RecycleMillis=1200000
    RecycleSet=MemberSet(Size=0
      )
    )
)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:3)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
        at com.tangosol.coherence.component.net.Cluster.startSystemServices(Cluster.CDB:4)
        at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:53)
        at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:12)
        at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:5)
        at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:10)
        at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:32)
        at com.tangosol.coherence.component.util.SafeCluster.getRunningCluster(SafeCluster.CDB:7)
        at com.tangosol.coherence.component.util.SafeCluster.getClusterName(SafeCluster.CDB:1)
        at com.tangosol.internal.metrics.MetricSupport.createMetricTags(MetricSupport.java:654)
        at com.tangosol.internal.metrics.MetricSupport.getMetrics(MetricSupport.java:269)
        at com.tangosol.internal.metrics.MetricSupport.registerInternal(MetricSupport.java:197)
        at com.tangosol.internal.metrics.MetricSupport.register(MetricSupport.java:139)
        at com.tangosol.coherence.component.net.management.Gateway.registerMetrics(Gateway.CDB:1)
        at com.tangosol.coherence.component.net.management.gateway.Remote.registerLocalModel(Remote.CDB:10)
        at com.tangosol.coherence.component.net.management.Gateway.register(Gateway.CDB:8)
        at com.tangosol.coherence.component.util.SafeService.register(SafeService.CDB:14)
        at com.tangosol.coherence.component.util.SafeService.ensureRunningService(SafeService.CDB:57)
        at com.tangosol.coherence.component.util.SafeService.startInternal(SafeService.CDB:14)
        at com.tangosol.coherence.component.util.SafeService$StartAction.run(SafeService.CDB:1)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
        at com.tangosol.net.security.DoAsAction.run(DoAsAction.java:42)
        at java.security.AccessController.doPrivileged(Native Method)
        at com.tangosol.coherence.component.util.SafeService.start(SafeService.CDB:4)
        at com.tangosol.net.ExtensibleConfigurableCacheFactory.startService(ExtensibleConfigurableCacheFactory.java:796)
        at com.tangosol.net.ExtensibleConfigurableCacheFactory.ensureService(ExtensibleConfigurableCacheFactory.java:659)
        at com.tangosol.coherence.config.scheme.AbstractCachingScheme.realizeCache(AbstractCachingScheme.java:53)
        at com.tangosol.net.ExtensibleConfigurableCacheFactory.ensureCacheInternal(ExtensibleConfigurableCacheFactory.java:340)
        at com.tangosol.net.ExtensibleConfigurableCacheFactory.ensureCache(ExtensibleConfigurableCacheFactory.java:213)
        at com.tangosol.net.ConfigurableCacheFactory.ensureTypedCache(ConfigurableCacheFactory.java:92)
        at com.tangosol.net.ConfigurableCacheFactory.ensureCache(ConfigurableCacheFactory.java:57)
        at oracle.mds.internal.cache.coherence.CoherenceCache.initNamedCache(CoherenceCache.java:655)
        at oracle.mds.internal.cache.coherence.CoherenceCache.<init>(CoherenceCache.java:117)
        at oracle.mds.internal.cache.coherence.CoherenceCacheProvider.<init>(CoherenceCacheProvider.java:59)
        at oracle.mds.internal.cache.CacheProviderFactory.getCacheProvider(CacheProviderFactory.java:95)
        at oracle.mds.internal.cache.CacheProviderFactory.getCacheProvider(CacheProviderFactory.java:52)
        at oracle.mds.core.MDSInstance.createCacheProvider(MDSInstance.java:1545)
        at oracle.mds.core.MDSInstance.initCache(MDSInstance.java:1898)
        at oracle.mds.core.MDSInstance.<init>(MDSInstance.java:3189)
        at oracle.mds.core.MDSInstance.createMDSInstance(MDSInstance.java:2441)
        at oracle.mds.core.MDSInstance.access$500(MDSInstance.java:464)
        at oracle.mds.core.MDSInstance$MDSInstanceStore$1.call(MDSInstance.java:4104)
        at oracle.mds.core.MDSInstance$MDSInstanceStore$1.call(MDSInstance.java:4082)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at oracle.mds.core.MDSInstance$MDSInstanceStore.getOrCreateMDSInstance(MDSInstance.java:4124)
        at oracle.mds.core.MDSInstance.getOrCreateInstanceInternal(MDSInstance.java:2949)
        at oracle.mds.core.MDSInstance.getOrCreateInstance(MDSInstance.java:918)
        at oracle.adf.share.config.ADFMDSConfig.createMDSInstanceFromelement(ADFMDSConfig.java:355)
        at oracle.adf.share.config.MDSConfigFactory.createApplication(MDSConfigFactory.java:119)
        at oracle.adf.share.config.ADFConfigFactory.findOrCreateADFConfig(ADFConfigFactory.java:178)
        at oracle.adf.share.config.ADFConfigFactory.findOrCreateADFConfig(ADFConfigFactory.java:108)
        at oracle.adf.share.config.ADFConfigFactory.findOrCreateADFConfig(ADFConfigFactory.java:85)
        at oracle.adf.share.ADFContext.getADFConfigEx(ADFContext.java:1028)
        at oracle.adf.share.http.ServletContextScopeAdapter.addScopeListener(ServletContextScopeAdapter.java:325)
        at oracle.adf.share.http.ServletContextScopeAdapter.getScope(ServletContextScopeAdapter.java:114)
        at oracle.adf.share.http.ServletADFContext.getApplicationScopeMap(ServletADFContext.java:417)
        at oracle.adf.share.ADFContext.getScope(ADFContext.java:543)
        at oracle.adf.share.ADFContext.getApplicationName(ADFContext.java:1591)
        at oracle.adf.share.http.ServletADFContext.getApplicationName(ServletADFContext.java:183)
        at oracle.adf.share.http.ServletADFContext.initialize(ServletADFContext.java:522)
        at oracle.adf.share.http.ServletADFContext.initThreadContext(ServletADFContext.java:444)
        at oracle.adf.mbean.share.config.ADFConfigLifeCycleCallBack.contextInitialized(ADFConfigLifeCycleCallBack.java:99)
        at weblogic.servlet.internal.EventsManager$FireContextListenerAction.run(EventsManager.java:719)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:344)
        at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)
        at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203)
        at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71)
        at weblogic.servlet.internal.EventsManager.executeContextListener(EventsManager.java:251)
        at weblogic.servlet.internal.EventsManager.notifyContextCreatedEvent(EventsManager.java:204)
        at weblogic.servlet.internal.EventsManager.notifyContextCreatedEvent(EventsManager.java:189)
        at weblogic.servlet.internal.WebAppServletContext.preloadResources(WebAppServletContext.java:1929)
        at weblogic.servlet.internal.WebAppServletContext.start(WebAppServletContext.java:3191)
        at weblogic.servlet.internal.WebAppModule.startContexts(WebAppModule.java:1864)
        at weblogic.servlet.internal.WebAppModule.start(WebAppModule.java:919)
        at weblogic.application.internal.ExtensibleModuleWrapper$StartStateChange.next(ExtensibleModuleWrapper.java:360)
        at weblogic.application.internal.ExtensibleModuleWrapper$StartStateChange.next(ExtensibleModuleWrapper.java:356)
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:45)
        at weblogic.application.internal.ExtensibleModuleWrapper.start(ExtensibleModuleWrapper.java:138)
        at weblogic.application.internal.flow.ModuleListenerInvoker.start(ModuleListenerInvoker.java:124)
        at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:233)
        at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:228)
        at weblogic.application.utils.StateMachineDriver$ParallelChange.run(StateMachineDriver.java:83)
        at weblogic.application.utils.StateMachineDriver.nextStateInParallel(StateMachineDriver.java:144)
        at weblogic.application.internal.flow.ModuleStateDriver.parallelStart(ModuleStateDriver.java:90)
        at weblogic.application.internal.flow.StartModulesFlow.activate(StartModulesFlow.java:44)
        at weblogic.application.internal.BaseDeployment$2.next(BaseDeployment.java:752)
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:45)
        at weblogic.application.internal.BaseDeployment.activate(BaseDeployment.java:262)
        at weblogic.application.internal.EarDeployment.activate(EarDeployment.java:66)
        at weblogic.application.internal.DeploymentStateChecker.activate(DeploymentStateChecker.java:165)
        at weblogic.deploy.internal.targetserver.AppContainerInvoker.activate(AppContainerInvoker.java:90)
        at weblogic.deploy.internal.targetserver.BasicDeployment.activate(BasicDeployment.java:274)
        at weblogic.deploy.internal.targetserver.BasicDeployment.activateFromServerLifecycle(BasicDeployment.java:507)
        at weblogic.management.deploy.internal.DeploymentAdapter$1.doActivate(DeploymentAdapter.java:53)
        at weblogic.management.deploy.internal.DeploymentAdapter.activate(DeploymentAdapter.java:202)
        at weblogic.management.deploy.internal.AppTransition$2.transitionAppInternal(AppTransition.java:53)
        at weblogic.management.deploy.internal.AppTransition.transitionApp(AppTransition.java:138)
        at weblogic.management.deploy.internal.ConfiguredDeployments$2.doItem(ConfiguredDeployments.java:741)
        at weblogic.management.deploy.internal.parallel.BucketInvoker.invoke(BucketInvoker.java:138)
        at weblogic.management.deploy.internal.ConfiguredDeployments.transitionAppsParallel(ConfiguredDeployments.java:749)
        at weblogic.management.deploy.internal.ConfiguredDeployments.transitionAppsParallel(ConfiguredDeployments.java:702)
        at weblogic.management.deploy.internal.ConfiguredDeployments.activate(ConfiguredDeployments.java:381)
        at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:233)
        at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:219)
        at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:133)
        at weblogic.server.AbstractServerService.postConstruct(AbstractServerService.java:76)
        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1287)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:333)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:375)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:305)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:85)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:116)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:698)
        at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:78)
        at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:212)
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:235)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:358)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:305)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:85)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:116)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:90)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1237)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1168)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:681)
        at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
        at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
        at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
        at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
        at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:655)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420)
        at weblogic.wofrk.ExecuteThread.run(ExecuteThread.java:360)
>
<Feb 17, 2021 9:54:39,097 PM UTC> <Warning> <Coherence> <BEA-000000> <2021-02-17 21:54:39.097/390.640 Oracle Coherence GE 12.2.1.4.0 <Warning> (thread=Cluster, member=n/a): This Member(Id=0, Timestamp=2021-02-17 21:54:08.625, Address=192.168.0.100:29705, MachineId=39780, Location=process:23944, Role=WeblogicServer) has been attempting to join the cluster at address /239.192.0.0:7574 with TTL 4 for 30 seconds without success; this could indicate a mis-configured TTL value, or it may simply be the result of a busy cluster or active failover.>
<Feb 17, 2021 9:57:15,763 PM UTC> <Warning> <Coherence> <BEA-000000> <2021-02-17 21:57:15.763/547.306 Oracle Coherence GE 12.2.1.4.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; multicast networking appears to be inoperable on interface 192.168.0.100 as this process isn't receiving even its own transmissions; consider forcing IPv4 via -Djava.net.preferIPv4Stack=true>
<Feb 17, 2021 9:57:22,575 PM UTC> <Warning> <Coherence> <BEA-000000> <2021-02-17 21:57:22.575/554.118 Oracle Coherence GE 12.2.1.4.0 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; multicast networking appears to be inoperable on interface 192.168.0.100 as this process isn't receiving even its own transmissions; consider forcing IPv4 via -Djava.net.preferIPv4Stack=true>
^C<Feb 17, 2021 9:57:24,338 PM UTC> <Notice> <WebLogicServer> <BEA-000388> <JVM called the WebLogic Server shutdown hook. The server will force shutdown now.>
<Feb 17, 2021 9:57:24,341 PM UTC> <Notice> <WebLogicServer> <BEA-000396> <Server shutdown has been requested by <WLS Kernel>.>
<Feb 17, 2021 9:57:24,348 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN.>
<Feb 17, 2021 9:57:29,356 PM UTC> <Notice> <Log Management> <BEA-170037> <The log monitoring service timer has been stopped.>
<Feb 17, 2021 9:57:29,432 PM UTC> <Warning> <JMX> <BEA-149513> <JMX Connector Server stopped at service:jmx:iiop://192.168.0.100:7101/jndi/weblogic.management.mbeanservers.edit.>
<Feb 17, 2021 9:57:29,432 PM UTC> <Warning> <JMX> <BEA-149513> <JMX Connector Server stopped at service:jmx:iiop://192.168.0.100:7101/jndi/weblogic.management.mbeanservers.domainruntime.>
<Feb 17, 2021 9:57:54,347 PM UTC> <Notice> <WebLogicServer> <BEA-000378> <The server failed to shut down within the configured timeout of 30 seconds. The server process will now exit.>
<Feb 17, 2021 9:57:54,366 PM UTC> <Error> <> <BEA-000000> <THREAD DUMP from JVM taken at 'Wed Feb 17 21:57:54 2021'

If you encounter the error above, you’re facing two issues at same time.

1. WebLogic is using IPv6 to reach internal components.
2. You may need to apply Coherence patch # 32124456.

Step One (Configure WebLogic to prefer IPv4 communication):

1) Navigate to $DOMAIN_HOME/bin.
2) Backup the setDomainEnv.sh.

$ cp setDomainEnv.sh setDomainEnv-`date '+%Y%m%d'`.sh

3) Open the setDomainEnv.sh in edit mode.
4) Edit the JAVA_OPTIONS and add the below parameters on setDomainEnv.sh:

-Djava.net.preferIPv6Addresses=false -Djava.net.preferIPv4Stack=true

Example:

JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.net.preferIPv6Addresses=false -Djava.net.preferIPv4Stack=true"
export JAVA_OPTIONS

# SET THE CLASSPATH

5) Save the file.
6) Now run the below command.

. ./setDomainEnv.sh 

# (Please observe the double dots in the command)

Reference: Coherence Member Failed to Join Multicast-Listener Cluster Intermittently With Error Delaying formation of a new cluster (Doc ID 2362920.1).

Step Two (Solving Coherence bug):

Apply the latest Coherence patch # 32124456—the downloadable link is available in the below document. Please search for 32124456.

$ORACLE_HOME/OPatch/opatch apply 122147 -jdk $JAVA_HOME

Fixed Bugs List With Patch Downloads—Oracle Coherence 12.2.1.4.0 for Java (Doc ID 2616220.1).

Step Three:

1) Navigate to $DOMAIN_HOME/servers/AdminServer.

cd $DOMAIN_HOME/servers/AdminServer

5) Save the file.
6) Now run the below command.

. ./setDomainEnv.sh (Please observe the double dots in the command)

Step Four:

7) Start the Admin Server.
8) After following the above steps, please check if the Admin Server has achieved RUNNING status.

cd DOMAIN_HOME/bin
./startWebLogic.sh

I hope this information is helpful!

If you have any questions or thoughts, please leave them in the comments.