Hi, I have seen several cases like mine here, but somehow the provided answers don't fit.
We have several SMD agents installed, some of them as local agents on windows machines for EEM.
But one agent on my local PC is always disconnecting since some time. I can't exactly tell, when this started. I know that it worked last December, but fails at least since end of August, 2015.
Symptom:
in dev_smdagent:
J Thu Sep 10 13:01:38 2015
J : [12624] 13:01:38 ***Warning: Thread.stop(): Thread.stop() called using java.lang.ThreadDeath in com.sap.engine.services.rmi_p4.P4ObjectBrokerClientImpl.close()V:264
F
F [Thr 764] Thu Sep 10 13:01:46 2015
F [Thr 764] *** LOG => State changed from 3 (Running) to 10 (Starting apps).
F [Thr 764] *** LOG state real time: 240.286 CPU time: 0.405 sys, 3.088 usr
F [Thr 764] *** LOG total real time: 10655.463 CPU time: 67.096 sys, 270.037 usr
F [Thr 764]
F
F [Thr 764] Thu Sep 10 13:01:54 2015
F [Thr 764] *** LOG => State changed from 10 (Starting apps) to 3 (Running).
F [Thr 764] *** LOG state real time: 8.281 CPU time: 0.826 sys, 2.948 usr
F [Thr 764] *** LOG total real time: 10663.744 CPU time: 67.922 sys, 272.986 usr
F [Thr 764]
this is repeating every five minutes or so.
Corresponding error message in the Agent-Errorlog on the Solman:
| Sep 10, 2015 12:53:17 PM | SMDloggerSystem | [ms://r0184i1:8101/P4] Ping failed (java.rmi.RemoteException) com.sap.engine.services.jndi.persistent.exceptions.NameNotFoundException: Object not found in lookup of C011A0170.group.rwe.com_DAA_SMDA98. |
This is the SMDSystem.log:
| Sep 10, 2015 1:02:46 PM [Thread[ExRun:OutsideDiscovery_1,5,OutsideD...] Info | Reading JStartup environment... | |
| Sep 10, 2015 1:02:46 PM [Thread[ExRun:OutsideDiscovery_1,5,OutsideD...] Info | 'SAPLOCALHOST' found, The value 'C011A0170' will be used to determine the Agent name. | |
| Sep 10, 2015 1:02:46 PM [Thread[ExRun:OutsideDiscovery_1,5,OutsideD...] Info | InetAddress Resolving for 'C011A0170' - FQN: C011A0170.group.rwe.com SQN: C011A0170 IP: 10.183.81.13 | |
| Sep 10, 2015 1:02:46 PM [Thread[ExRun:OutsideDiscovery_1,5,OutsideD...] Info | InetAddress Resolving for localhost - FQN: C011A0170.group.rwe.com SQN: C011A0170 IP: 10.183.81.13 | |
| Sep 10, 2015 1:05:17 PM [Thread[Connector,5,main] | ] Error | [ms://r0184i1:8101/P4] Ping failed (java.rmi.RemoteException) java.rmi.RemoteException: Agent not known on server. |
| Sep 10, 2015 1:05:17 PM [Thread[ExRun:default_1,5,default:ExecTG] | ] Info | [J2EEClusterConnector.forgetAccessPoint] Connection status changed to DISCONNECTED. Forgotting all access points. |
| Sep 10, 2015 1:05:17 PM [Thread[Connector,5,main] | ] Info | [SMDConnector.resetBroker] reset the p4 broker (close:true) |
| Sep 10, 2015 1:05:17 PM [Thread[Connector,5,main] | ] Info | [SMDConnector.resetBroker] p4 broker closed. |
And this is from the smdagent_trace file:
#1.5 #020085E2EC040001000006EE00001D5400051F6293326155#1441883120815#SMDloggerSystem##SMDloggerSystem#######Thread[Connector,5,main]##0#0#Info##Plain###Old Agent Handle com.sap.smd.agent.AgentHandle@5b1e96 cleanup.#
#1.5 #020085E2EC040001000006EF00001D5400051F629332B139#1441883120836#SMDloggerSystem##SMDloggerSystem#######Thread[Connector,5,main]##0#0#Info##Plain###Reading JStartup environment...#
#1.5 #020085E2EC040001000006F000001D5400051F629332B1A0#1441883120836#SMDloggerSystem##SMDloggerSystem#######Thread[Connector,5,main]##0#0#Info##Plain###'SAPLOCALHOST' found, The value 'C011A0170' will be used to determine the Agent name.#
#1.5 #020085E2EC040001000006F100001D5400051F629332B208#1441883120836#SMDloggerSystem##SMDloggerSystem#######Thread[Connector,5,main]##0#0#Info##Plain###InetAddress Resolving for 'C011A0170' - FQN: C011A0170.group.rwe.com SQN: C011A0170 IP: 10.183.81.13#
#1.5 #020085E2EC040001000006F200001D5400051F629332B330#1441883120836#SMDloggerSystem##SMDloggerSystem#######Thread[Connector,5,main]##0#0#Info##Plain###InetAddress Resolving for localhost - FQN: C011A0170.group.rwe.com SQN: C011A0170 IP: 10.183.81.13#
#1.5 #020085E2EC040001000006F300001D5400051F629332B3C4#1441883120836#SMDloggerSystem##SMDloggerSystem#######Thread[Connector,5,main]##0#0#Info##Plain###Agent Handle have been created#
#1.5 #020085E2EC040001000006F400001D5400051F629332B4A0#1441883120837#SMDloggerSystem##SMDloggerSystem#######Thread[Connector,5,main]##0#0#Info##Plain### Agent Version .............. : 7.10.12.2.20141128090136#
#1.5 #020085E2EC040001000006F500001D5400051F629332B4F5#1441883120837#SMDloggerSystem##SMDloggerSystem#######Thread[Connector,5,main]##0#0#Info##Plain### Agent ID ................... : C011A0170.group.rwe.com_DAA_SMDA98#
#1.5 #020085E2EC040001000006F600001D5400051F629332B53E#1441883120837#SMDloggerSystem##SMDloggerSystem#######Thread[Connector,5,main]##0#0#Info##Plain### Assigned Server Name ....... : rob_r002358#
#1.5 #020085E2EC040001000006F700001D5400051F629332B58B#1441883120837#SMDloggerSystem##SMDloggerSystem#######Thread[Connector,5,main]##0#0#Info##Plain### Canonical Host Name ........ : C011A0170.group.rwe.com#
#1.5 #020085E2EC040001000006F800001D5400051F629332B5CD#1441883120837#SMDloggerSystem##SMDloggerSystem#######Thread[Connector,5,main]##0#0#Info##Plain###[ms://r0184i1:8101/P4] Registering agent on server ... #
#1.5 #020085E2EC0400E40000000000001D5400051F6293474752#1441883122185#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###[AgentContext.stopApplications] Stopping Agent applications ...#
#1.5 #020085E2EC0400E40000000100001D5400051F6293474F02#1441883122187#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###[ExecutionManager.Shutdown] Shutting down ExecutionManager{selfd:filter}.#
#1.5 #020085E2EC0400E40000000200001D5400051F62934751CF#1441883122188#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Warning##Plain###Cannot close the thread group selfd:filter#
#1.5 #020085E2EC0400E40000000300001D5400051F629356FDEA#1441883123214#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###[ExecutionManager.Shutdown] Shutting down ExecutionManager{e2emai:jobmgr}.#
#1.5 #020085E2EC0400E40000000400001D5400051F6293570E9F#1441883123219#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Warning##Plain###[com.sap.smd.om.impl.broker.ObjectReference@2c8d79 (entities:com.sap.smd.agent.application.e2emai.MAIService [ref])]: unload did not unregister all dependencies of entities:com.sap.smd.agent.application.e2emai.MAIService#
#1.5 #020085E2EC0400E40000000500001D5400051F6293570F7F#1441883123219#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Warning##Plain###[com.sap.smd.om.impl.broker.ObjectReference@1cb215e (states:dpc.startup [ref])]: unload did not unregister dependant "entities:com.sap.smd.agent.application.e2emai.MAIService" of "states:dpc.startup" - could be cycle. Will clean dependants set now!#
#1.5 #020085E2EC0400E40000000600001D5400051F6293582B0F#1441883123292#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###[ExecutionManager.Shutdown] Shutting down ExecutionManager{dpc:job}.#
#1.5 #020085E2EC0400E40000000700001D5400051F6293583E69#1441883123297#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###the deamon thread group dpc:job is already closed.#
#1.5 #020085E2EC0400E40000000800001D5400051F6293583F03#1441883123297#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###the thread group dpc:job is already closed.#
#1.5 #020085E2EC0400E40000000900001D5400051F62935842C5#1441883123298#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###[ExecutionManager.Shutdown] Shutting down ExecutionManager{check}.#
#1.5 #020085E2EC0400E40000000A00001D5400051F6293585A3B#1441883123304#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###the deamon thread group check is already closed.#
#1.5 #020085E2EC0400E40000000B00001D5400051F6293585AD1#1441883123304#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###the thread group check is already closed.#
#1.5 #020085E2EC0400E40000000C00001D5400051F6293586C0B#1441883123308#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###[ExecutionManager.Shutdown] Shutting down ExecutionManager{agelet_WilyEM}.#
#1.5 #020085E2EC0400E40000000D00001D5400051F629358713E#1441883123310#com.sap.smd.e2edcc##com.sap.smd.e2edcc#######Thread[Thread-628,5,main]##0#0#Info##Plain###[DCCService.clean] DCCService cleaned.#
#1.5 #020085E2EC0400E40000000E00001D5400051F62935871CE#1441883123310#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###[ExecutionManager.Shutdown] Shutting down ExecutionManager{OutsideDiscovery}.#
#1.5 #020085E2EC0400E40000000F00001D5400051F62935874E5#1441883123311#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###the deamon thread group OutsideDiscovery is already closed.#
#1.5 #020085E2EC0400E40000001000001D5400051F629358751F#1441883123311#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###the thread group OutsideDiscovery is already closed.#
#1.5 #020085E2EC0400E40000001100001D5400051F629358758B#1441883123311#com.sap.smd.e2edcc##com.sap.smd.e2edcc#######Thread[Thread-628,5,main]##0#0#Info##Plain###[DCCService.dispose] DCCService released.#
#1.5 #020085E2EC0400E40000001200001D5400051F6293587661#1441883123311#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###[WHS] Shutting down Wily Handler Service#
#1.5 #020085E2EC0400E40000001300001D5400051F62935883ED#1441883123314#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Warning##Plain###[com.sap.smd.om.impl.broker.ObjectReference@14d1e30 (applications:com.sap.smd.agent.application.e2emai [ref])]: trying to unload dependent "entities:com.sap.smd.agent.application.e2emai.MAIService" revealed dependency to non-existent broker#
#1.5 #020085E2EC0400E40000001400001D5400051F6293588587#1441883123315#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Warning##Plain###[com.sap.smd.om.impl.broker.ObjectReference@c11070 (applications:com.sap.smd.agent.application.nodescontroller [ref])]: trying to unload dependent "entities:com.sap.smd.agent.application.nodescontroller.nodesController" revealed dependency to non-existent broker#
#1.5 #020085E2EC0400E40000001500001D5400051F6293588821#1441883123316#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Warning##Plain###[com.sap.smd.om.impl.broker.ObjectReference@13853f7 (applications:com.sap.smd.agent.application.runtime [ref])]: trying to unload dependent "entities:com.sap.smd.agent.application.runtime.servicesEntityFactory" revealed dependency to non-existent broker#
#1.5 #020085E2EC0400E40000001600001D5400051F629358893A#1441883123316#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###[ExecutionManager.Shutdown] Shutting down ExecutionManager{WILYHOST_SCHEDULER}.#
#1.5 #020085E2EC0400E40000001700001D5400051F6293588CF6#1441883123317#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###the deamon thread group WILYHOST_SCHEDULER is already closed.#
#1.5 #020085E2EC0400E40000001800001D5400051F6293588D5E#1441883123317#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###the thread group WILYHOST_SCHEDULER is already closed.#
#1.5 #020085E2EC0400E40000001900001D5400051F6293588DA3#1441883123317#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###[ExecutionManager.Shutdown] Shutting down ExecutionManager{EEM_SCHEDULER}.#
#1.5 #020085E2EC0400E40000001A00001D5400051F6293588F4B#1441883123317#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###the deamon thread group EEM_SCHEDULER is already closed.#
#1.5 #020085E2EC0400E40000001B00001D5400051F6293588FA1#1441883123317#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###the thread group EEM_SCHEDULER is already closed.#
#1.5 #020085E2EC0400E40000001C00001D5400051F6293588FDF#1441883123317#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###[ExecutionManager.Shutdown] Shutting down ExecutionManager{com.sap.maimgr}.#
#1.5 #020085E2EC0400E40000001D00001D5400051F6293589086#1441883123318#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###[ExecutionManager.Shutdown] Shutting down ExecutionManager{com.sap.exmmgr}.#
#1.5 #020085E2EC0400E40000001E00001D5400051F62935890D5#1441883123318#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###[AgentContext.stopApplications] Agent Applications stopped.#
#1.5 #020085E2EC0400E40000001F00001D5400051F62938A1C1E#1441883126564#SMDloggerSystem##SMDloggerSystem#######Thread[Thread-628,5,main]##0#0#Info##Plain###[AgentContext.startApplications] Starting Agent Applications...#
And what I did: I re-registerd the SMD agent with
smdsetup managingconf hostname:"sapms://r0184i1.rwe.com" port:"8101" user:"*******" pwd:"*****"
several times (without error), restarted the SAP hostagent and the SMD agent, but every five minutes or so the agent restarts and disappears from the list of available agents.
Connection method for all robot pc agents is MS/P4, since our solution manager is clustered. All agents are 32bit, Version 7.10.12.2
Any suggestions?
thanks for your advice in advance ...