2025-09-01 Mon 02:09:23:609.893 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:23:662.671 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:23:662.828 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692563662555 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:23:662.849 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:23:662.861 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:23:662.874 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692563662709 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 42851 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:23:662.880 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:23:662.997 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:23:663.058 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:23:663.102 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:23:663.125 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692563663090 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:23:663.133 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:24:366.459 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:24:366.515 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692564366323 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:24:366.526 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:24:366.698 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:24:366.716 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692564366688 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 37995 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:24:366.736 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:24:367.026 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:24:367.077 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:24:367.157 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:24:367.170 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692564367128 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:24:367.176 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:24:572.296 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:24:572.525 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692564572267 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:24:572.544 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:24:572.557 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:24:572.572 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692564572461 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 35541 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:24:572.580 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:24:572.765 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:24:572.816 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:24:572.865 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:24:572.897 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692564572855 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:24:572.905 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:26:138.392 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:138.448 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692566138360 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:26:138.459 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:26:138.851 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:138.874 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692566138842 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 44821 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:26:138.880 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:26:139.261 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:26:139.359 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:26:139.408 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:139.444 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692566139393 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:26:139.471 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:26:139.482 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:139.495 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692566139401 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:26:139.503 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:26:139.512 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:139.545 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692566139472 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:26:139.577 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:26:139.596 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:139.620 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692566139512 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:09:26:139.630 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:26:139.958 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:139.978 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692566139948 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:26:140.003 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:26:140.013 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:140.047 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692566139973 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:26:140.064 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:26:140.012 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:09:26:185.178 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:185.240 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692566185137 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:26:185.253 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:26:185.310 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:185.335 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692566185304 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 42485 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:26:185.341 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:26:185.629 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:26:185.696 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:26:185.738 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:185.760 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692566185726 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:26:185.783 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:26:607.189 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:607.256 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692566607042 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:26:607.267 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:26:607.276 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:607.288 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692566607204 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 45633 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:26:607.294 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:26:607.569 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:26:607.617 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:26:607.671 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:607.690 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692566607664 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:26:607.696 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:26:891.361 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:891.415 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692566891332 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:26:891.424 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:26:891.913 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:891.960 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692566891885 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 34279 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:26:891.969 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:26:892.240 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:26:892.319 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:26:892.366 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:892.550 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692566892357 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:26:892.570 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:26:892.583 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:892.598 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692566892365 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:26:892.607 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:26:892.616 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:892.645 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692566892429 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:26:892.660 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:26:892.786 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:892.828 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692566892774 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:26:892.847 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:26:892.855 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:892.870 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692566892791 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:09:26:892.877 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:26:893.364 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:893.386 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692566893324 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:26:893.395 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:26:893.403 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:893.406 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:09:26:893.422 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692566893372 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:26:893.512 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:26:938.517 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:938.576 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692566938480 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:26:938.586 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:26:938.926 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:938.949 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692566938913 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 35383 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:26:938.956 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:26:939.264 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:26:939.323 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:26:939.368 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:26:939.406 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692566939352 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:26:939.415 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:27:248.051 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:27:248.107 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692567247916 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:27:248.129 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:27:248.707 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:27:248.732 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692567248686 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 38339 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:27:248.742 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:27:249.282 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:27:249.424 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:27:249.487 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:27:249.517 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692567249460 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:27:249.529 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:27:983.581 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:27:983.640 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692567983552 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:27:983.650 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:27:984.006 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:27:984.056 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692567983977 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 35543 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:27:984.066 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:27:984.345 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:27:984.747 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:27:984.789 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:27:984.806 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692567984778 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:27:984.813 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:28:363.376 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:28:363.447 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692568363340 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:28:363.460 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:28:363.938 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:28:363.978 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692568363896 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 35641 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:28:363.997 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:28:364.200 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:28:364.264 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:28:364.308 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:28:364.332 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692568364298 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:28:364.340 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:28:364.348 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:28:364.357 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692568364303 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:28:364.375 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:28:364.392 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:28:364.416 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692568364386 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:28:364.427 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:28:364.445 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:28:364.463 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692568364440 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:09:28:364.470 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:28:365.149 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:28:365.178 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692568365071 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:28:365.186 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:28:365.205 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:28:365.224 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692568365134 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:28:365.233 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:28:365.258 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:09:28:412.549 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:28:412.612 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692568412383 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:28:412.625 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:28:413.188 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:28:413.236 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692568413144 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 39771 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:28:413.249 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:28:413.449 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:28:413.501 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:28:413.558 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:28:413.589 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692568413548 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:28:413.595 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:29:157.666 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:29:157.750 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692569157626 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:29:157.766 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:29:158.310 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:29:158.332 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692569158277 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 42709 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:29:158.340 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:29:158.644 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:29:158.772 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:29:158.829 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:29:158.842 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692569158820 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:29:158.849 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:29:322.786 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:29:322.851 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692569322679 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:29:322.861 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:29:323.286 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:29:323.318 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692569323264 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 33061 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:29:323.326 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:29:323.551 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:29:323.618 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:29:323.689 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:29:323.719 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692569323672 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:29:323.725 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:29:785.263 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:29:785.335 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692569785057 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:29:785.345 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:29:785.353 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:29:785.364 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692569785286 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 35601 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:29:785.370 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:29:785.707 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:29:785.753 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:29:785.845 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:29:785.860 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692569785815 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:29:785.867 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:30:639.947 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:30:640.062 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692570639816 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:30:640.075 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:30:640.086 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:30:640.098 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692570639968 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 34869 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:30:640.105 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:30:640.273 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:30:640.318 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:30:640.371 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:30:640.501 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692570640362 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:30:640.523 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:30:640.534 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:30:640.546 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692570640366 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:30:640.555 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:30:640.581 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:30:640.611 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692570640420 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:30:640.625 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:30:641.265 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:30:641.306 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692570640702 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:30:641.316 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:30:641.324 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:30:641.338 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692570640719 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:09:30:641.351 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:30:641.357 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:30:641.367 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692570641227 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:30:641.373 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:30:641.379 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:30:641.392 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692570641252 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:30:641.399 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:30:641.446 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:09:30:694.594 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:30:694.653 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692570694503 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:30:694.663 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:30:695.214 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:30:695.239 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692570695205 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 41441 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:30:695.245 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:30:695.532 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:30:695.651 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:30:695.706 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:30:695.725 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692570695685 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:30:695.732 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:31:268.043 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:31:268.107 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692571267750 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:31:268.118 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:31:268.126 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:31:268.138 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692571268007 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 41987 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:31:268.144 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:31:268.362 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:31:268.462 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:31:268.562 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:31:268.584 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692571268539 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:31:268.592 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:31:614.009 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:31:614.077 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692571613835 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:31:614.091 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:31:614.582 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:31:614.625 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692571614457 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 46041 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:31:614.634 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:31:614.889 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:31:614.942 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:31:614.994 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:31:615.048 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692571614974 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:31:615.057 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:33:263.165 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:263.234 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692573262971 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:33:263.248 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:33:263.625 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:263.654 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692573263594 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 45521 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:33:263.665 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:33:263.934 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:33:264.083 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:33:264.159 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:264.203 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692573264119 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:33:264.213 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:33:264.222 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:264.232 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692573264137 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:33:264.238 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:33:264.245 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:264.267 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692573264200 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:33:264.287 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:33:264.295 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:264.310 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692573264259 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:09:33:264.316 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:33:265.016 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:265.032 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692573264967 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:33:265.039 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:33:265.046 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:265.065 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692573265040 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:33:265.074 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:33:265.358 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:09:33:314.549 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:314.607 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692573314502 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:33:314.618 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:33:315.151 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:315.191 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692573315074 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 46355 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:33:315.202 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:33:315.366 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:33:315.424 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:33:315.479 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:315.501 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692573315469 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:33:315.508 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:33:457.955 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:458.023 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692573457918 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:33:458.032 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:33:458.351 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:458.364 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692573458342 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 34161 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:33:458.382 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:33:458.709 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:33:458.848 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:33:458.892 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:458.931 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692573458875 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:33:458.940 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:33:458.948 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:458.956 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692573458880 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:33:458.962 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:33:458.970 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:459.000 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692573458926 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:33:459.010 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:33:459.026 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:459.049 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692573459021 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:33:459.058 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:33:459.073 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:459.086 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692573459036 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:09:33:459.092 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:33:459.527 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:459.547 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692573459516 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:33:459.554 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:33:459.566 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:459.584 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692573459561 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:33:459.597 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:33:459.674 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:09:33:510.843 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:510.902 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692573510805 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:33:510.912 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:33:511.374 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:511.423 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692573511340 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 33879 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:33:511.434 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:33:511.662 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:33:511.775 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:33:511.830 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:511.844 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692573511820 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:33:511.861 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:33:703.307 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:703.375 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692573703275 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:33:703.386 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:33:703.810 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:703.853 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692573703787 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 41297 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:33:703.862 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:33:704.092 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:33:704.143 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:33:704.191 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:33:704.211 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692573704181 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:33:704.217 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:34:215.657 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:34:215.717 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692574215618 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:34:215.727 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:34:216.195 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:34:216.236 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692574216174 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 33911 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:34:216.256 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:34:216.466 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:34:216.560 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:34:216.643 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:34:216.659 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692574216607 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:34:216.666 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:34:448.001 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:34:448.061 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692574447956 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:34:448.071 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:34:448.614 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:34:448.656 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692574448586 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 44705 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:34:448.666 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:34:448.846 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:34:448.893 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:34:448.939 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:34:448.958 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692574448929 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:34:448.990 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:35:084.833 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:35:084.887 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692575084698 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:35:084.896 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:35:084.906 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:35:084.918 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692575084851 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 38983 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:35:084.925 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:35:085.156 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:35:085.208 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:35:085.265 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:35:085.917 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692575085255 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:35:085.928 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:35:085.937 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:35:085.946 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692575085259 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:35:085.951 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:35:085.957 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:35:085.997 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692575085318 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:35:086.017 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:35:086.027 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:35:086.041 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692575085363 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:09:35:086.047 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:35:086.053 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:35:086.061 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692575085821 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:35:086.067 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:35:086.073 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:35:086.086 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692575085870 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:35:086.097 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:35:085.892 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:09:35:133.023 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:35:133.078 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692575132996 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:35:133.087 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:35:133.473 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:35:133.488 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692575133457 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 37939 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:35:133.500 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:35:133.821 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:35:134.102 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:35:134.145 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:35:134.184 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692575134128 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:35:134.195 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:36:107.005 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:36:107.075 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692576106847 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:36:107.086 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:36:107.094 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:36:107.177 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692576107014 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 46637 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:36:107.187 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:36:107.471 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:36:107.542 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:36:107.617 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:36:107.631 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692576107608 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:36:107.637 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:36:160.339 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:36:160.404 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692576160309 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:36:160.415 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:36:160.471 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:36:160.483 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692576160463 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 34025 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:36:160.505 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:36:160.799 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:36:160.849 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:36:160.903 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:36:160.928 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692576160895 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:36:160.934 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:36:339.192 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:36:339.250 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692576339091 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:36:339.261 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:36:339.626 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:36:339.651 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692576339617 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 45581 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:36:339.658 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:36:339.934 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:36:340.032 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:36:340.068 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:36:340.084 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692576340060 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:36:340.091 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:37:664.433 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:37:664.558 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692577664274 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 20 dequeues = 20 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 5341 dequeues = 5341 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 21 dequeues = 21 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 21 dequeues = 21 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 5405 dequeues = 5405 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756692548697708 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756692576340065 ] max_sm_queue_count = 7 ] 2025-09-01 Mon 02:09:37:664.577 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:09:37:664.590 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:37:664.767 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692577664400 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1065 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 97 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1212 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 291 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4111 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1174 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 2680, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 5341, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:09:37:664.785 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:09:37:670.467 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:37:670.527 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692577670437 [SYS_NOTICE]: UveVrouterAgent: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 xmpp_peer_list= [ [ [ ip = 10.20.0.129 status = 0 setup_time = 0 primary = 1 ], [ ip = 10.20.0.25 status = 0 setup_time = 0 primary = 0 ], ] ] dns_servers= [ [ (*_iter149) = 10.20.0.25, (*_iter149) = 10.20.0.193, ] ] ] 2025-09-01 Mon 02:09:37:670.538 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: UveVrouterAgent 2025-09-01 Mon 02:09:37:670.558 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:37:670.693 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692577670479 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 in_tpkts = 1539 in_bytes = 188416 out_tpkts = 1621 out_bytes = 90806 exception_packets = 55 exception_packets_allowed = 43 raw_vhost_stats= [ name = vhost0 in_pkts = 1539 in_bytes = 188416 out_pkts = 1621 out_bytes = 90806 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 1 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 1 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 1 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 1 ] total_in_bandwidth_utilization = 6.93889e-15 total_out_bandwidth_utilization = 1.61655e-14 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens6 _iter260->second = 1280, ] ] phy_band_out_bps= [ [ _iter261->first = ens6 _iter261->second = 2982, ] ] raw_phy_if_stats= [ [ _iter262->first = ens6 [ in_pkts = 1453 in_bytes = 83750 out_pkts = 1541 out_bytes = 188468 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens6 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 02:09:37:670.708 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 02:09:37:670.717 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:37:670.745 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692577670518 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 raw_xmpp_stats= [ [ _iter306->first = 10.20.0.129 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], _iter306->first = 10.20.0.25 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 02:09:37:670.752 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 02:09:37:700.378 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:37:700.436 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692577700349 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:37:700.447 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:37:700.828 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:37:700.853 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692577700820 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 37955 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:37:700.859 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:37:701.110 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:37:701.308 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:37:701.346 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:37:701.367 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692577701334 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:37:701.391 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:37:701.401 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:37:701.414 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692577701339 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:37:701.424 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:37:701.433 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:37:701.463 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692577701387 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:37:701.477 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:37:701.498 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:37:701.517 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692577701485 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:37:701.530 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:37:701.537 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:37:701.552 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692577701521 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:09:37:701.558 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:37:702.101 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:37:702.126 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692577702053 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:37:702.134 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:37:702.141 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:37:702.173 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692577702087 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:37:702.183 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:37:702.210 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:09:37:750.415 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:37:750.473 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692577750385 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:37:750.484 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:37:750.999 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:37:751.048 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692577750945 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 43753 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:37:751.060 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:37:752.775 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:37:753.137 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:37:753.221 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:37:753.251 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692577753206 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:37:753.260 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:38:087.130 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:38:087.204 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692578087083 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:38:087.218 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:38:087.535 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:38:087.568 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692578087525 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 39527 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:38:087.595 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:38:087.962 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:38:088.116 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:38:088.173 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:38:088.192 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692578088161 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:38:088.200 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:38:697.790 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 02:09:38:698.524 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Connect : Start Connect timer 10.0.0.38:8086 2025-09-01 Mon 02:09:38:698.641 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:09:38:698.669 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692578697878 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:38:698.681 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:38:698.691 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:09:38:698.809 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692578698626 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1069 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 99 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1215 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 292 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4122 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1177 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 2687, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 5361, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:09:38:698.835 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:09:38:698.918 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 02:09:38:698.927 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: -(-1):EvTcpConnectFail 2025-09-01 Mon 02:09:38:698.949 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 02:09:38:698.958 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Connect : EvTcpConnectFail 2025-09-01 Mon 02:09:38:699.459 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:38:699.494 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Idle 2025-09-01 Mon 02:09:38:699.535 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:38:699.666 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692578699027 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.131:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1070 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 100 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1215 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 292 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4122 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1177 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 2687, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 5363, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:09:38:699.684 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:09:38:699.696 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:38:699.713 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692578699056 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:38:699.721 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:38:699.727 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 02:09:38:699.754 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:38:699.871 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692578699526 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.131:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1070 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 100 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1216 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 292 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4122 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1177 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 2688, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 5363, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:09:38:699.885 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:09:38:734.387 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:38:734.447 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692578734348 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:38:734.471 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:38:734.802 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:38:734.824 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692578734794 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 45001 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:38:734.831 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:38:735.144 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:38:735.252 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:38:735.289 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:38:735.305 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692578735279 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:38:735.312 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:39:779.713 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:779.772 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692579779676 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:39:779.782 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:39:780.078 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:780.092 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692579780068 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 35537 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:39:780.098 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:39:780.414 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:39:780.655 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:39:780.694 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:780.710 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692579780683 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:39:780.716 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:39:780.723 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:780.731 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692579780689 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:39:780.737 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:39:780.743 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:780.765 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692579780737 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:39:780.775 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:39:780.787 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:780.802 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692579780783 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:09:39:780.819 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:39:781.270 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:781.285 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692579781238 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:39:781.292 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:39:781.298 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:781.315 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692579781283 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:39:781.327 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:39:781.467 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:09:39:831.672 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:831.730 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692579831635 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:39:831.740 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:39:832.080 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:832.103 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692579832070 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 34593 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:39:832.110 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:39:832.431 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:39:832.738 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:39:832.784 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:832.799 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692579832776 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:39:832.806 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:39:940.275 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:940.331 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692579940245 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:39:940.342 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:39:940.432 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:940.453 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692579940425 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 36565 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:39:940.460 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:39:940.698 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:39:940.760 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:39:940.813 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:940.838 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692579940801 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:39:940.846 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:39:940.854 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:940.862 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692579940806 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:39:940.867 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:39:940.889 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:940.911 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692579940883 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:39:940.921 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:39:940.973 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:941.008 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692579940968 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:39:941.029 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:39:941.043 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:941.059 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692579940993 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:09:39:941.065 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:39:941.531 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:941.553 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692579941474 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:39:941.560 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:39:941.567 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:941.586 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692579941509 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:39:941.601 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:39:941.636 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:09:39:992.772 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:992.828 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692579992733 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:39:992.839 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:39:993.321 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:993.361 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692579993290 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 35105 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:39:993.370 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:39:993.540 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:39:993.590 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:39:993.666 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:39:993.686 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692579993624 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:39:993.696 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:40:661.546 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:40:661.620 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692580661513 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:40:661.634 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:40:661.997 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:40:662.030 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692580661952 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 39849 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:40:662.037 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:40:662.324 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:40:662.432 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:40:662.479 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:40:662.493 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692580662471 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:40:662.500 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:40:751.071 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:40:751.128 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692580750932 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:40:751.138 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:40:751.147 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:40:751.159 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692580751103 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 34413 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:40:751.165 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:40:751.394 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:40:751.441 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:40:751.504 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:40:751.550 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692580751496 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:40:751.557 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:40:893.945 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:40:894.020 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692580893783 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:40:894.031 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:40:894.040 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:40:894.050 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692580893938 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 46757 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:40:894.056 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:40:894.255 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:40:894.313 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:40:894.368 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:40:894.387 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692580894361 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:40:894.393 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:42:120.368 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:42:120.426 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692582120338 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:42:120.436 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:42:120.534 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:42:120.585 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692582120526 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 33403 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:42:120.593 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:42:120.892 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:42:120.960 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:42:121.027 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:42:121.054 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692582121005 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:42:121.062 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:42:121.071 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:42:121.080 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692582121021 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:42:121.086 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:42:121.116 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:42:121.144 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692582121108 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:42:121.168 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:42:121.189 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:42:121.217 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692582121176 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:09:42:121.226 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:42:121.726 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:09:42:121.764 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:42:121.785 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692582121667 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:42:121.792 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:42:121.799 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:42:121.818 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692582121709 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:42:121.832 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:42:174.933 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:42:175.124 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692582174826 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:42:175.134 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:42:175.430 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:42:175.448 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692582175410 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 35075 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:42:175.456 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:42:175.724 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:42:175.799 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:42:175.873 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:42:175.897 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692582175860 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:42:175.904 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:42:856.575 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:42:856.630 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692582856537 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:42:856.639 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:42:856.708 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:42:856.731 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692582856700 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 38025 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:42:856.750 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:42:857.025 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:42:857.109 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:42:857.154 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:42:857.178 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692582857143 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:42:857.186 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:42:875.824 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:42:875.869 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692582875698 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:42:875.879 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:42:875.888 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:42:875.900 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692582875852 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 39213 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:42:875.906 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:42:876.169 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:42:876.217 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:42:876.279 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:42:876.310 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692582876270 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:42:876.333 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:43:112.130 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:43:112.187 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692583111997 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:43:112.197 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:43:112.205 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:43:112.216 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692583112152 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 41197 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:43:112.221 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:43:112.468 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:43:112.528 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:43:112.598 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:43:112.613 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692583112576 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:43:112.620 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:44:298.762 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:44:298.815 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692584298649 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:44:298.824 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:44:299.660 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:44:299.685 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692584299649 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 34413 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:44:299.704 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:44:300.020 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:44:300.062 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:44:300.586 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:44:300.628 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692584300087 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:44:300.636 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:44:300.643 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:44:300.652 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692584300092 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:44:300.657 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:44:300.664 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:44:300.685 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692584300168 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:44:300.695 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:44:300.703 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:44:300.737 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692584300257 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:44:300.748 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:44:300.757 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:44:300.774 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692584300272 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:09:44:300.780 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:44:300.787 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:44:300.795 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692584300764 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:44:300.801 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:44:300.808 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:44:300.822 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692584300788 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:44:300.832 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:09:44:300.832 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:44:351.010 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:44:351.070 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692584350964 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:44:351.079 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:44:351.568 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:44:351.607 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692584351542 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 37411 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:44:351.616 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:44:351.850 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:44:351.907 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:44:351.962 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:44:351.997 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692584351952 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:44:352.016 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:45:218.814 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:45:218.988 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692585218782 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:45:218.999 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:45:219.008 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:45:219.020 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692585218936 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 39733 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:45:219.026 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:45:219.236 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:45:219.299 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:45:219.369 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:45:219.400 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692585219360 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:45:219.407 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:45:324.116 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:45:324.167 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692585324080 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:45:324.177 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:45:324.458 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:45:324.482 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692585324450 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 44601 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:45:324.525 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:45:328.619 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:45:328.758 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:45:328.854 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:45:328.867 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692585328788 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:45:328.874 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:46:565.389 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:46:565.453 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692586565286 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:46:565.466 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:46:566.391 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:46:566.425 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692586566371 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 36935 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:46:566.433 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:46:566.779 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:46:566.870 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:46:566.952 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:46:566.966 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692586566908 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:46:566.999 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:46:567.007 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:46:567.015 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692586566914 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:46:567.021 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:46:567.752 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:46:567.784 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692586567059 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:46:567.795 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:46:567.803 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:46:567.818 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692586567139 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:46:567.826 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:46:567.843 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:46:567.857 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692586567159 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:09:46:567.863 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:46:567.871 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:46:567.880 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692586567682 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:46:567.886 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:46:567.892 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:46:567.905 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692586567711 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:46:567.912 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:46:567.938 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:09:46:616.161 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:46:616.219 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692586616037 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:46:616.235 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:46:616.247 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:46:616.264 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692586616187 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 45023 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:46:616.273 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:46:616.541 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:46:616.598 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:46:616.655 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:46:616.668 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692586616646 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:46:616.674 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:47:147.039 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:47:147.091 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692587146921 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:47:147.101 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:47:147.557 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:47:147.577 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692587147540 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 44275 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:47:147.584 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:47:147.807 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:47:147.881 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:47:147.944 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:47:148.004 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692587147934 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:47:148.011 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:47:160.423 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:47:160.481 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692587160387 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:47:160.529 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:47:160.852 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:47:160.863 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692587160847 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 40779 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:47:160.878 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:47:161.083 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:47:161.145 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:47:161.181 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:47:161.199 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692587161169 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:47:161.206 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:47:161.233 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:47:161.242 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692587161174 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:47:161.248 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:47:161.254 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:47:161.276 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692587161222 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:47:161.286 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:47:161.327 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:47:161.342 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692587161263 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:09:47:161.348 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:47:161.711 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:47:161.730 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692587161703 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:47:161.746 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:47:161.757 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:47:161.785 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692587161728 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:47:161.794 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:47:161.822 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:09:47:211.040 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:47:211.099 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692587211003 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:47:211.109 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:47:211.694 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:47:211.734 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692587211664 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 36033 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:47:211.742 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:47:212.007 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:47:212.068 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:47:212.114 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:47:212.143 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692587212103 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:47:212.152 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:47:606.874 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:47:606.938 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692587606841 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:47:606.953 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:47:607.046 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:47:607.090 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692587607035 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 39695 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:47:607.099 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:47:607.421 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:47:607.512 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:47:607.569 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:47:607.589 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692587607549 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:47:607.597 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:48:274.312 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:48:275.282 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:48:276.194 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692588274277 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:48:276.274 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:48:276.286 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:48:276.300 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692588274807 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 35011 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:48:276.307 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:48:276.416 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:48:276.464 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:48:276.477 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692588276454 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:48:276.483 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:49:179.567 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:49:179.640 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692589179526 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:49:179.654 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:49:180.122 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:49:180.164 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692589180092 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 46455 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:49:180.173 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:49:180.384 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:49:180.466 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:49:180.545 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:49:180.561 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692589180532 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:49:180.578 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:49:180.586 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:49:180.594 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692589180544 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:49:180.600 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:49:181.516 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:09:49:181.696 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:49:181.743 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692589180658 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:49:181.758 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:49:181.769 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:49:181.789 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692589180727 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:09:49:181.804 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:49:181.813 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:49:181.910 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692589181423 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:49:181.920 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:49:181.931 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:49:181.958 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692589181487 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:49:181.970 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:49:226.652 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:49:226.711 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692589226615 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:49:226.722 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:49:226.892 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:49:226.909 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692589226883 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 34693 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:49:226.926 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:49:227.374 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:49:227.463 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:49:227.516 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:49:227.536 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692589227498 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:49:227.544 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:49:497.691 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:49:497.767 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692589497653 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:49:497.783 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:49:497.842 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:49:497.875 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692589497833 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 46271 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:49:497.886 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:49:498.154 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:49:498.224 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:49:498.285 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:49:498.309 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692589498274 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:49:498.317 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:50:148.644 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:50:148.708 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692590148612 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:50:148.722 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:50:148.781 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:50:148.812 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692590148772 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 37661 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:50:148.822 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:50:149.098 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:50:149.183 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:50:149.223 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:50:149.246 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692590149212 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:50:149.254 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:50:172.786 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:50:172.838 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692590172639 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:50:172.848 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:50:172.868 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:50:172.880 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692590172806 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 35791 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:50:172.887 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:50:173.120 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:50:173.169 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:50:173.224 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:50:173.246 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692590173215 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:50:173.253 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:51:252.203 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:51:252.267 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692591252073 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:51:252.279 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:51:252.702 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:51:252.724 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692591252679 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 44415 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:51:252.731 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:51:253.018 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:51:253.083 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:51:253.165 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:51:253.370 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692591253149 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:51:253.382 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:51:253.393 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:51:253.407 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692591253158 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:51:253.415 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:51:253.424 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:51:253.465 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692591253229 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:51:253.480 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:51:253.601 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:51:253.634 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692591253589 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:51:253.651 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:51:253.661 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:51:253.675 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692591253637 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:09:51:253.682 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:51:254.232 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:51:254.266 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692591254163 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:51:254.276 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:51:254.286 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:51:254.314 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692591254213 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:51:254.337 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:51:254.371 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:09:51:300.597 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:51:300.650 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692591300525 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:51:300.659 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:51:301.091 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:51:301.138 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692591301071 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 41429 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:51:301.147 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:51:301.398 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:51:301.477 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:51:301.540 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:51:301.558 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692591301513 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:51:301.565 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:52:081.395 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:52:081.457 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692592081358 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:52:081.467 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:52:082.026 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:52:082.079 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692592081969 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 34861 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:52:082.094 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:52:082.330 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:52:082.387 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:52:082.444 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:52:082.461 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692592082421 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:52:082.468 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:52:201.682 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:52:201.744 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692592201644 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:52:201.755 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:52:202.253 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:52:202.301 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692592202233 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 43745 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:52:202.310 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:52:202.528 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:52:202.583 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:52:202.648 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:52:202.685 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692592202638 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:52:202.691 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:53:710.516 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:53:710.575 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692593710390 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:53:710.586 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:53:710.994 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:53:711.015 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692593710961 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 44859 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:53:711.023 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:53:711.260 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:53:711.325 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:53:711.387 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:53:711.547 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692593711377 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:53:711.556 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:53:711.564 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:53:711.574 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692593711386 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:53:711.579 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:53:711.596 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:53:711.623 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692593711438 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:53:711.635 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:53:712.524 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:53:712.563 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692593711748 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:53:712.575 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:53:712.584 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:53:712.601 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692593711772 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:09:53:712.615 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:53:712.622 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:53:712.633 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692593712548 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:53:712.640 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:53:712.647 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:53:712.664 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692593712579 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:53:712.674 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:53:712.679 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:09:53:762.893 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:53:762.949 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692593762862 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:53:762.959 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:53:763.440 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:53:763.486 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692593763415 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 40505 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:53:763.495 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:53:763.688 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:53:763.747 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:53:763.791 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:53:763.805 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692593763783 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:53:763.812 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:54:145.428 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:54:145.479 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692594145324 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:54:145.489 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:54:145.795 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:54:145.823 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692594145778 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 40601 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:54:145.831 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:54:146.205 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:54:146.306 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:54:146.357 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:54:146.403 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692594146342 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:54:146.412 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:54:146.420 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:54:146.433 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692594146348 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:54:146.442 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:54:146.458 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:54:146.491 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692594146450 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:54:146.505 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:54:146.522 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:54:146.545 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692594146510 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:09:54:146.559 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:54:147.044 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:54:147.067 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692594146949 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:54:147.074 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:54:147.081 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:54:147.161 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692594147031 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:54:147.177 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:54:147.221 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:09:54:199.427 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:54:199.482 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692594199324 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:54:199.491 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:54:199.862 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:54:199.914 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692594199835 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 37439 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:54:199.940 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:54:200.154 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:54:200.219 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:54:200.265 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:54:200.284 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692594200254 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:54:200.291 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:54:290.883 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:54:290.935 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692594290770 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:54:290.944 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:54:291.412 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:54:291.439 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692594291396 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 44455 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:54:291.446 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:54:291.688 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:54:291.754 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:54:291.817 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:54:291.864 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692594291807 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:54:291.882 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:54:817.020 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:54:817.071 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692594816896 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:54:817.081 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:54:817.484 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:54:817.515 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692594817461 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 40645 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:54:817.538 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:54:817.795 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:54:817.863 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:54:817.927 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:54:817.973 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692594817917 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:54:817.992 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:55:172.486 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:55:172.555 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692595172431 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:55:172.565 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:55:172.885 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:55:172.899 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692595172874 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 32769 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:55:172.917 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:55:173.191 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:55:173.318 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:55:173.369 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:55:173.409 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692595173347 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:55:173.419 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:55:790.614 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:55:790.672 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692595790574 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:55:790.682 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:55:791.204 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:55:791.251 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692595791178 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 42021 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:55:791.261 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:55:791.472 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:55:791.528 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:55:791.563 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:55:791.578 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692595791555 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:55:791.600 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:55:791.610 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:55:791.623 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692595791561 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:55:791.631 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:55:791.640 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:55:791.664 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692595791633 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:55:791.676 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:55:791.688 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:55:791.709 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692595791678 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:09:55:791.718 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:55:792.341 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:55:792.363 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692595792308 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:55:792.371 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:55:792.379 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:55:792.401 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692595792331 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:55:792.411 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:55:792.480 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:09:55:842.232 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:55:842.284 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692595842107 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:55:842.294 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:55:842.702 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:55:842.726 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692595842688 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 44145 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:55:842.733 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:55:843.057 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:55:843.127 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:55:843.177 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:55:843.195 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692595843159 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:55:843.203 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:56:853.850 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:56:853.941 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692596853577 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:56:853.970 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:56:855.354 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:56:855.400 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692596855295 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 42829 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:56:855.413 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:56:855.870 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:56:855.997 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:56:856.076 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:56:856.103 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692596856065 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:56:856.117 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:56:887.322 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:56:887.414 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692596887282 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:56:887.429 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:56:887.667 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:56:887.690 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692596887657 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 45837 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:56:887.700 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:56:888.095 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:56:888.302 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:56:888.358 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:56:888.398 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692596888335 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:56:888.407 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:57:063.584 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:57:063.765 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692597063550 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:57:063.780 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:57:063.789 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:57:063.806 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692597063707 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 33239 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:57:063.812 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:57:064.023 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:57:064.072 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:57:064.127 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:57:064.158 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692597064120 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:57:064.164 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:58:072.013 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:58:072.074 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692598071962 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:58:072.085 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:58:072.599 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:58:072.647 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692598072562 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 37967 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:58:072.656 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:58:072.868 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:58:072.950 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:58:073.002 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:58:073.016 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692598072991 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:58:073.023 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:58:073.030 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:58:073.050 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692598072998 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:58:073.056 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:58:073.063 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:58:073.093 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692598073050 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:58:073.104 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:58:073.154 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:58:073.187 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692598073147 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:58:073.196 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:58:073.203 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:58:073.217 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692598073163 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:09:58:073.226 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:58:073.709 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:58:073.725 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692598073672 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:58:073.732 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:58:073.739 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:58:073.747 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:09:58:073.756 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692598073726 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:09:58:073.836 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:09:58:126.912 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:58:126.990 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692598126785 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:58:127.001 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:58:127.014 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:58:127.031 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692598126939 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 39055 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:58:127.039 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:58:127.288 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:58:127.343 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:58:127.410 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:58:127.431 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692598127391 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:58:127.441 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:58:796.540 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:58:796.587 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692598796472 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:58:796.598 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:58:797.029 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:58:797.121 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692598797015 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 46115 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:58:797.129 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:58:797.373 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:58:797.448 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:58:797.505 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:58:797.524 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692598797484 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:58:797.531 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:59:162.684 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:59:162.749 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692599162547 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:59:162.763 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:59:162.775 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:59:162.788 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692599162705 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 34755 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:59:162.794 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:09:59:163.131 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:09:59:163.179 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:09:59:163.237 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:09:59:163.257 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692599163229 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:09:59:163.263 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:00:744.318 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:744.380 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692600744210 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:00:744.390 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:00:744.808 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:744.834 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692600744790 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 42775 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:00:744.841 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:00:745.100 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:00:745.187 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:00:745.313 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:745.326 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692600745301 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:00:745.332 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:00:745.339 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:745.347 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692600745313 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:00:745.353 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:00:745.410 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:745.453 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692600745398 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:00:745.498 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:00:745.511 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:745.535 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692600745484 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:00:745.547 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:00:745.556 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:745.577 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692600745500 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:00:745.587 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:00:746.074 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:746.103 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692600746004 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:00:746.110 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:00:746.117 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:746.136 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692600746046 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:00:746.148 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:00:746.199 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:00:795.452 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:795.510 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692600795344 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:00:795.521 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:00:796.226 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:796.246 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692600796213 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 40727 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:00:796.253 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:00:796.591 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:00:797.305 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:00:797.400 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:797.414 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692600797343 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:00:797.421 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:00:844.368 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:844.420 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692600844266 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:00:844.430 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:00:844.787 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:844.806 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692600844773 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 33581 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:00:844.813 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:00:845.084 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:00:845.155 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:00:845.239 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:846.063 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692600845224 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:00:846.074 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:00:846.083 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:846.092 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692600845233 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:00:846.108 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:00:846.114 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:846.136 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692600845304 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:00:846.146 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:00:846.178 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:846.191 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692600845364 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:00:846.197 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:00:846.207 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:846.215 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692600845879 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:00:846.223 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:00:846.229 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:846.242 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692600845937 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:00:846.250 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:00:845.959 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:00:899.284 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:899.347 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692600899118 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:00:899.359 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:00:899.369 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:899.381 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692600899323 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 34753 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:00:899.388 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:00:899.638 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:00:899.710 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:00:899.777 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:00:899.801 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692600899766 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:00:899.809 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:01:215.615 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:01:215.677 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692601215472 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:01:215.688 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:01:216.319 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:01:216.360 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692601216271 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 45777 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:01:216.371 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:01:216.685 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:01:216.740 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:01:216.785 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:01:216.955 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692601216774 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:01:216.966 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:01:778.550 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:01:778.610 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692601778477 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:01:778.621 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:01:778.698 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:01:778.716 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692601778687 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 42093 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:01:778.804 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:01:779.055 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:01:779.128 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:01:779.189 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:01:779.214 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692601779176 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:01:779.222 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:01:935.016 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:01:935.090 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692601934970 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:01:935.104 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:01:935.176 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:01:935.197 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692601935166 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 41937 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:01:935.206 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:01:935.504 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:01:935.625 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:01:935.699 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:01:935.719 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692601935687 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:01:935.727 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:02:613.795 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:02:613.859 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692602613644 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:02:613.870 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:02:613.879 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:02:613.891 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692602613813 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 43507 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:02:613.897 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:02:614.118 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:02:614.164 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:02:614.220 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:02:614.241 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692602614212 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:02:614.247 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:02:614.254 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:02:614.262 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692602614216 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:02:614.268 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:02:614.295 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:02:614.318 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692602614289 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:02:614.328 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:02:614.346 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:02:614.360 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692602614341 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:02:614.366 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:02:614.939 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:02:614.954 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692602614908 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:02:614.960 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:02:614.999 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:02:615.039 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692602614971 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:02:615.050 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:02:615.420 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:02:664.664 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:02:664.720 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692602664628 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:02:664.730 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:02:665.077 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:02:665.116 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692602665060 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 40721 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:02:665.124 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:02:665.351 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:02:665.420 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:02:665.587 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:02:665.600 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692602665472 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:02:665.619 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:03:705.504 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:03:705.562 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692603705349 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:03:705.573 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:03:705.582 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:03:705.595 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692603705514 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 45235 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:03:705.601 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:03:706.166 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:03:706.247 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:03:706.309 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:03:706.348 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692603706296 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:03:706.356 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:03:709.544 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:03:709.577 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692603709527 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:03:709.586 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:03:709.660 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:03:709.684 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692603709654 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 37043 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:03:709.690 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:03:709.882 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:03:709.958 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:03:710.018 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:03:710.037 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692603710001 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:03:710.046 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:03:843.859 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:03:843.917 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692603843818 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:03:843.928 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:03:844.263 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:03:844.279 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692603844253 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 36605 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:03:844.294 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:03:844.569 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:03:844.660 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:03:844.703 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:03:844.741 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692603844693 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:03:844.760 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:05:285.024 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:05:285.078 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692605284901 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:05:285.088 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:05:285.389 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:05:285.410 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692605285371 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 40597 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:05:285.417 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:05:285.700 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:05:285.785 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:05:285.842 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:05:285.887 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692605285813 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:05:285.897 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:05:285.906 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:05:285.918 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692605285818 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:05:285.925 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:05:285.966 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:05:286.004 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692605285883 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:05:286.028 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:05:286.037 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:05:286.055 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692605285953 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:05:286.062 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:05:286.073 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:05:286.086 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692605285967 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:05:286.095 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:05:286.497 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:05:286.520 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692605286453 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:05:286.527 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:05:286.534 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:05:286.553 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692605286484 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:05:286.563 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:05:286.808 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:05:336.966 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:05:337.041 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692605336859 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:05:337.054 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:05:337.664 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:05:337.682 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692605337655 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 43271 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:05:337.746 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:05:338.009 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:05:338.045 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:05:338.118 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:05:338.129 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692605338075 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:05:338.136 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:05:636.170 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:05:636.232 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692605636076 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:05:636.242 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:05:636.908 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:05:636.930 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692605636897 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 34797 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:05:636.937 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:05:637.185 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:05:637.219 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:05:637.289 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:05:637.300 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692605637245 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:05:637.324 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:06:265.297 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:06:265.351 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692606265200 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:06:265.361 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:06:265.652 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:06:265.669 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692606265639 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 33155 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:06:265.676 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:06:265.993 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:06:266.443 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:06:266.514 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:06:266.558 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692606266469 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:06:266.568 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:07:552.907 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:552.959 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692607552808 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:07:552.968 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:07:553.292 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:553.321 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692607553273 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 42853 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:07:553.346 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:07:553.611 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:07:553.699 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:07:553.754 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:553.798 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692607553727 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:07:553.807 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:07:553.823 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:553.839 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692607553731 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:07:553.848 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:07:553.858 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:553.886 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692607553781 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:07:553.898 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:07:553.913 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:553.937 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692607553823 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:07:553.944 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:07:554.358 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:554.385 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692607554310 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:07:554.395 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:07:554.405 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:554.433 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692607554346 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:07:554.447 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:07:555.050 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:07:558.427 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:558.455 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692607558367 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:07:558.464 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:07:558.824 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:558.838 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692607558817 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 42389 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:07:558.845 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:07:559.111 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:07:559.145 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:07:559.818 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:559.843 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692607559166 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:07:559.853 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:07:559.863 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:559.874 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692607559170 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:07:559.882 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:07:559.891 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:559.920 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692607559270 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:07:559.937 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:07:559.945 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:559.959 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692607559355 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:07:559.966 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:07:559.973 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:559.995 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692607559369 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:07:560.001 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:07:560.010 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:560.018 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692607559779 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:07:560.023 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:07:560.029 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:560.042 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692607559798 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:07:560.049 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:07:560.091 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:07:600.305 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:600.356 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692607600204 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:07:600.368 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:07:600.863 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:600.909 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692607600828 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 44275 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:07:600.918 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:07:601.192 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:07:601.296 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:07:601.331 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:601.350 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692607601322 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:07:601.357 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:07:612.309 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:612.355 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692607612188 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:07:612.363 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:07:612.811 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:612.830 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692607612798 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 43795 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:07:612.837 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:07:613.179 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:07:613.248 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:07:613.320 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:613.360 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692607613310 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:07:613.366 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:07:664.720 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:664.817 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692607664569 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 21 dequeues = 21 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 5672 dequeues = 5672 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 22 dequeues = 22 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 22 dequeues = 22 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 5739 dequeues = 5739 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756692578699493 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756692607613318 ] max_sm_queue_count = 7 ] 2025-09-01 Mon 02:10:07:664.831 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:10:07:664.839 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:665.076 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692607664696 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.131:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1134 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 102 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1285 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 310 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4365 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1246 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 2844, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 5672, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:10:07:665.106 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:10:07:670.757 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:670.801 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692607670650 [SYS_NOTICE]: UveVrouterAgent: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 xmpp_peer_list= [ [ [ ip = 10.20.0.193 status = 0 setup_time = 0 primary = 1 ], [ ip = 10.20.0.129 status = 0 setup_time = 0 primary = 0 ], ] ] dns_servers= [ [ (*_iter149) = 10.20.0.193, (*_iter149) = 10.20.0.25, ] ] ] 2025-09-01 Mon 02:10:07:670.810 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: UveVrouterAgent 2025-09-01 Mon 02:10:07:670.825 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:670.858 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692607670695 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 2230346464 used = 32524 free = 0 buffers = 0 cached = 2303930040 node_type = ] meminfo= [ virt = 4294967295 peakvirt = 18275337 res = 0 ] cpuload= [ one_min_avg = 0.095 five_min_avg = 0.105 fifteen_min_avg = 0.125 ] cpu_share = 1.525 ] ] 2025-09-01 Mon 02:10:07:670.868 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 02:10:07:670.879 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:670.897 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692607670700 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 cpu_info= [ [ [ mem_virt = 4294967295 cpu_share = 1.525 used_sys_mem = 1991069780 one_min_cpuload = 0.095 mem_res = 0 ], ] ] ] 2025-09-01 Mon 02:10:07:670.907 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 02:10:07:670.917 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:671.058 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692607670723 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 in_tpkts = 1616 in_bytes = 193854 out_tpkts = 1697 out_bytes = 94814 exception_packets = 58 exception_packets_allowed = 46 raw_vhost_stats= [ name = vhost0 in_pkts = 1616 in_bytes = 193854 out_pkts = 1697 out_bytes = 94814 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 1 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 1 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 1 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 1 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens6 [ in_pkts = 1526 in_bytes = 87632 out_pkts = 1618 out_bytes = 193906 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens6 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 02:10:07:671.074 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 02:10:07:671.082 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:07:671.105 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692607670745 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 raw_xmpp_stats= [ [ _iter306->first = 10.20.0.129 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], _iter306->first = 10.20.0.193 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 02:10:07:671.112 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 02:10:08:102.709 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:08:102.761 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692608102600 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:08:102.771 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:08:103.206 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:08:103.235 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692608103192 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 42661 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:08:103.242 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:08:103.546 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:08:103.621 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:08:103.710 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:08:103.724 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692608103675 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:08:103.731 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:08:618.523 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:08:618.579 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692608618420 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:08:618.588 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:08:619.020 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:08:619.044 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692608619001 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 36225 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:08:619.052 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:08:619.325 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:08:619.406 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:08:619.511 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:08:619.535 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692608619465 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:08:619.561 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:08:648.546 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:08:648.601 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692608648474 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:08:648.612 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:08:649.037 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:08:649.082 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692608649011 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 36287 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:08:649.091 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:08:649.331 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:08:649.379 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:08:649.438 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:08:649.475 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692608649427 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:08:649.486 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:08:699.578 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 02:10:08:700.181 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Connect : Start Connect timer 10.0.0.131:8086 2025-09-01 Mon 02:10:08:700.287 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:10:08:700.314 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692608699660 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.131:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:08:700.335 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:08:700.345 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:10:08:700.444 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 02:10:08:700.452 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: -(-1):EvTcpConnectFail 2025-09-01 Mon 02:10:08:700.546 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692608700273 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.131:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1134 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 104 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1288 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 310 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4374 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1249 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 2850, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 5688, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:10:08:700.564 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:10:08:700.577 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 02:10:08:700.583 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Connect : EvTcpConnectFail 2025-09-01 Mon 02:10:08:700.662 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:08:700.695 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Idle 2025-09-01 Mon 02:10:08:700.734 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:08:700.847 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692608700620 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1135 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 105 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1288 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 310 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4374 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1249 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 2850, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 5690, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:10:08:700.862 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:10:08:700.870 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:08:700.886 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692608700643 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:08:700.894 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:08:700.900 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 02:10:08:700.924 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:08:701.040 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692608700726 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1135 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 105 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1289 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 310 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4374 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1249 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 2851, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 5690, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:10:08:701.058 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:10:09:345.528 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:09:345.610 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692609345386 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:09:345.626 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:09:345.932 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:09:345.954 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692609345918 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 44509 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:09:345.961 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:09:346.215 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:09:346.355 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:09:346.428 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:09:346.471 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692609346400 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:09:346.481 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:09:346.489 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:09:346.499 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692609346407 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:09:346.517 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:09:346.524 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:09:346.549 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692609346468 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:09:346.559 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:09:346.568 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:09:346.584 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692609346545 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:09:346.590 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:09:347.174 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:09:347.198 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692609347124 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:09:347.212 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:09:347.219 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:09:347.311 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:09:347.603 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692609347163 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:09:347.621 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:09:397.499 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:09:397.551 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692609397465 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:09:397.562 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:09:397.697 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:09:397.709 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692609397673 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 45325 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:09:397.715 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:09:397.965 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:09:398.036 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:09:398.111 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:09:398.127 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692609398081 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:09:398.134 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:10:415.356 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:10:415.409 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692610415252 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:10:415.419 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:10:416.240 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:10:416.264 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692610416213 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 45763 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:10:416.272 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:10:416.529 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:10:416.596 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:10:416.660 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:10:416.672 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692610416624 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:10:416.679 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:10:521.585 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:10:521.643 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692610521552 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:10:521.665 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:10:521.734 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:10:521.757 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692610521725 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 43851 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:10:521.764 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:10:522.057 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:10:522.171 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:10:522.226 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:10:522.248 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692610522205 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:10:522.256 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:10:599.796 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:10:599.850 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692610599634 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:10:599.860 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:10:599.869 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:10:599.881 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692610599817 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 35611 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:10:599.888 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:10:600.119 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:10:600.168 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:10:600.232 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:10:600.259 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692610600224 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:10:600.265 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:12:135.903 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:12:136.185 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692612135869 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:12:136.200 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:12:136.213 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:12:136.231 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692612136037 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 40297 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:12:136.241 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:12:136.342 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:12:136.395 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:12:136.453 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:12:136.480 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692612136442 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:12:136.509 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:12:136.521 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:12:136.534 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692612136447 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:12:136.542 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:12:136.816 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:12:136.859 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692612136789 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:12:136.873 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:12:136.903 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:12:136.929 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692612136897 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:12:136.940 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:12:136.950 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:12:136.992 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692612136914 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:12:137.005 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:12:137.398 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:12:137.414 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692612137390 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:12:137.420 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:12:137.427 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:12:137.446 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692612137417 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:12:137.454 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:12:137.456 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:12:186.657 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:12:186.712 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692612186622 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:12:186.721 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:12:187.241 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:12:187.282 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692612187174 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 46781 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:12:187.290 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:12:187.513 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:12:187.569 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:12:187.653 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:12:187.669 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692612187617 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:12:187.676 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:12:288.809 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:12:288.868 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692612288770 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:12:288.879 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:12:289.373 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:12:289.414 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692612289339 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 46803 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:12:289.423 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:12:289.869 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:12:290.004 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:12:290.064 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:12:290.084 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692612290050 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:12:290.092 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:13:141.909 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:13:141.962 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692613141780 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:13:141.972 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:13:142.950 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:13:142.986 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692613142920 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 43273 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:13:142.994 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:13:143.246 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:13:143.334 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:13:143.382 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:13:143.410 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692613143368 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:13:143.421 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:14:272.466 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:272.571 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692614272361 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:14:272.581 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:14:272.731 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:272.776 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692614272704 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 33621 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:14:272.786 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:14:273.068 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:14:273.116 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:14:273.188 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:273.204 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692614273149 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:14:273.211 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:14:273.218 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:273.227 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692614273165 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:14:273.232 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:14:273.304 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:273.331 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692614273296 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:14:273.350 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:14:273.359 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:273.374 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692614273339 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:14:273.380 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:14:273.844 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:273.863 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692614273810 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:14:273.872 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:14:273.879 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:273.904 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692614273852 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:14:273.916 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:14:273.951 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:14:322.065 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:322.127 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692614322037 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:14:322.139 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:14:322.650 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:322.688 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692614322618 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 44535 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:14:322.696 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:14:322.892 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:14:322.949 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:14:323.008 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:323.028 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692614322999 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:14:323.035 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:14:806.460 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:806.523 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692614806359 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:14:806.533 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:14:807.315 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:807.347 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692614807298 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 43197 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:14:807.367 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:14:807.759 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:14:807.854 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:14:807.944 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:807.962 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692614807889 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:14:807.972 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:14:807.991 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:808.003 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692614807894 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:14:808.012 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:14:808.022 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:808.054 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692614807964 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:14:808.067 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:14:808.111 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:808.138 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692614808102 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:14:808.150 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:14:808.161 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:808.183 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692614808123 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:14:808.191 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:14:808.657 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:808.680 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692614808619 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:14:808.688 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:14:808.696 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:14:808.709 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:808.992 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692614808655 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:14:809.006 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:14:858.958 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:859.020 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692614858819 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:14:859.030 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:14:859.081 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:859.092 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692614859000 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 38495 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:14:859.098 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:14:859.491 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:14:859.541 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:14:859.605 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:14:859.864 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692614859589 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:14:859.882 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:15:195.639 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:15:195.690 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692615195532 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:15:195.700 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:15:196.124 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:15:196.145 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692615196100 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 35327 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:15:196.153 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:15:196.462 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:15:196.698 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:15:196.757 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:15:196.771 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692615196734 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:15:196.778 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:15:268.320 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:15:268.375 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692615268183 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:15:268.385 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:15:268.393 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:15:268.404 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692615268343 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 35439 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:15:268.422 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:15:268.682 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:15:268.736 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:15:268.804 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:15:268.837 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692615268795 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:15:268.844 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:15:840.858 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:15:840.913 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692615840737 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:15:840.924 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:15:841.244 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:15:841.262 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692615841230 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 46065 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:15:841.276 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:15:841.594 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:15:841.689 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:15:841.748 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:15:841.768 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692615841730 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:15:841.786 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:16:142.294 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:16:142.353 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692616142259 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:16:142.363 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:16:142.912 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:16:142.950 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692616142880 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 36315 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:16:142.959 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:16:143.171 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:16:143.216 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:16:143.261 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:16:143.894 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:16:144.018 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692616143252 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:16:144.027 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:16:144.044 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:16:144.054 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692616143259 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:16:144.059 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:16:144.074 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:16:144.095 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692616143333 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:16:144.106 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:16:144.113 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:16:144.128 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692616143391 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:16:144.134 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:16:144.140 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:16:144.148 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692616143842 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:16:144.154 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:16:144.160 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:16:144.173 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692616143873 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:16:144.183 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:16:197.047 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:16:197.121 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692616197007 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:16:197.136 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:16:197.637 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:16:197.675 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692616197602 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 38901 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:16:197.683 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:16:198.102 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:16:198.172 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:16:198.228 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:16:198.250 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692616198217 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:16:198.260 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:17:269.405 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:17:269.465 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692617269365 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:17:269.480 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:17:269.935 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:17:269.990 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692617269918 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 36857 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:17:270.000 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:17:270.361 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:17:270.488 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:17:270.527 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:17:270.545 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692617270516 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:17:270.552 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:17:302.980 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:17:303.040 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692617302936 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:17:303.050 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:17:303.564 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:17:303.579 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692617303555 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 38141 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:17:303.586 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:17:303.944 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:17:304.065 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:17:304.111 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:17:304.147 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692617304094 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:17:304.155 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:17:786.029 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:17:786.099 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692617785893 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:17:786.114 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:17:786.462 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:17:786.491 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692617786430 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 43273 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:17:786.501 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:17:786.726 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:17:786.825 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:17:786.888 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:17:786.929 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692617786860 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:17:786.939 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:18:796.957 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:18:797.017 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692618796858 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:18:797.027 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:18:797.719 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:18:797.745 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692618797707 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 34875 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:18:797.753 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:18:798.045 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:18:798.104 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:18:798.146 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:18:798.171 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692618798132 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:18:798.179 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:18:798.187 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:18:798.196 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692618798136 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:18:798.202 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:18:798.211 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:18:798.232 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692618798206 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:18:798.259 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:18:798.314 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:18:798.331 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692618798280 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:18:798.339 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:18:798.346 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:18:798.359 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692618798295 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:18:798.366 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:18:798.806 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:18:798.830 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692618798770 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:18:798.837 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:18:798.843 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:18:798.862 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692618798823 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:18:798.869 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:18:798.870 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:18:845.041 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:18:845.104 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692618845003 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:18:845.117 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:18:845.169 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:18:845.208 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692618845154 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 33619 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:18:845.217 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:18:845.491 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:18:845.561 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:18:845.610 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:18:845.633 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692618845599 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:18:845.640 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:19:124.705 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:19:124.767 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692619124676 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:19:124.776 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:19:124.865 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:19:124.894 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692619124857 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 35213 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:19:124.902 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:19:125.158 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:19:125.229 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:19:125.276 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:19:125.301 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692619125265 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:19:125.309 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:19:183.425 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.20.0.192:8085::10.20.0.193:52158 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 02:10:19:192.665 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.20.0.192:8085::10.20.0.193:52172 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 02:10:19:772.776 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:19:772.832 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692619772741 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:19:772.842 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:19:773.303 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:19:773.344 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692619773279 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 44643 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:19:773.352 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:19:773.673 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:19:773.782 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:19:773.822 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:19:773.841 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692619773811 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:19:773.849 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:21:048.299 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:048.384 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692621048269 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:21:048.394 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:21:048.439 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:048.453 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692621048430 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 34381 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:21:048.475 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:21:048.716 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:21:048.754 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:21:049.101 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:049.146 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692621048780 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:21:049.156 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:21:049.164 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:049.173 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692621048788 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:21:049.179 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:21:049.185 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:049.206 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692621048837 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:21:049.217 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:21:049.235 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:049.250 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692621048878 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:21:049.257 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:21:049.378 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:049.387 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692621049370 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:21:049.393 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:21:049.409 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:049.424 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692621049403 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:21:049.432 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:21:049.553 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:21:101.795 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:101.879 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692621101756 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:21:101.891 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:21:101.940 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:101.963 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692621101932 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 33453 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:21:101.969 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:21:102.233 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:21:102.296 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:21:102.393 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:102.416 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692621102362 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:21:102.427 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:21:772.179 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:772.475 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692621772135 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:21:772.505 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:21:772.518 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:772.536 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692621772272 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 41339 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:21:772.618 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:21:772.816 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:21:773.127 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:21:773.141 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:773.157 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692621772851 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:21:773.166 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:21:927.056 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:927.113 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692621927026 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:21:927.124 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:21:927.204 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:927.218 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692621927196 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 39883 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:21:927.224 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:21:927.500 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:21:927.565 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:21:927.609 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:927.640 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692621927597 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:21:927.652 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:21:927.678 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:927.691 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692621927603 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:21:927.699 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:21:927.709 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:927.755 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692621927676 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:21:927.770 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:21:927.782 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:927.805 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692621927753 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:21:927.816 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:21:927.826 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:927.851 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692621927767 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:21:927.859 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:21:928.279 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:928.303 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692621928225 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:21:928.310 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:21:928.319 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:928.345 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692621928258 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:21:928.358 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:21:928.388 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:21:980.607 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:980.661 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692621980569 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:21:980.672 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:21:981.138 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:981.183 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692621981111 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 39023 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:21:981.193 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:21:981.367 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:21:981.413 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:21:981.457 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:21:981.484 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692621981447 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:21:981.491 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:22:146.507 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:22:146.567 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692622146472 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:22:146.577 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:22:146.892 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:22:146.906 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692622146885 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 32837 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:22:146.913 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:22:147.200 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:22:147.290 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:22:147.341 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:22:147.359 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692622147323 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:22:147.366 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:22:899.726 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:22:899.781 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692622899590 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:22:899.792 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:22:899.801 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:22:899.814 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692622899745 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 40657 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:22:899.820 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:22:900.011 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:22:900.060 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:22:900.116 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:22:900.134 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692622900110 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:22:900.140 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:23:337.564 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:23:337.620 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692623337424 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:23:337.631 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:23:337.640 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:23:337.651 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692623337581 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 42531 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:23:337.657 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:23:337.944 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:23:338.004 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:23:338.060 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:23:338.077 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692623338054 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:23:338.083 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:23:338.090 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:23:338.098 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692623338059 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:23:338.103 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:23:338.132 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:23:338.157 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692623338125 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:23:338.183 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:23:338.194 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:23:338.216 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692623338187 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:23:338.225 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:23:338.650 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:23:338.669 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692623338613 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:23:338.681 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:23:338.688 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:23:338.705 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692623338648 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:23:338.720 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:23:338.671 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:23:383.859 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:23:383.918 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692623383823 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:23:383.929 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:23:384.328 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:23:384.354 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692623384315 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 46575 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:23:384.361 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:23:384.687 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:23:384.776 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:23:384.822 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:23:384.846 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692623384811 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:23:384.854 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:24:145.602 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:24:145.656 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692624145460 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:24:145.680 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:24:145.688 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:24:145.701 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692624145624 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 37331 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:24:145.707 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:24:145.926 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:24:145.985 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:24:146.041 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:24:146.072 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692624146033 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:24:146.078 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:24:330.062 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:24:330.186 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692624330013 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:24:330.197 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:24:330.206 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:24:330.218 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692624330182 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 41699 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:24:330.224 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:24:330.487 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:24:330.563 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:24:330.621 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:24:330.639 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692624330613 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:24:330.646 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:24:898.385 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:24:898.438 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692624898265 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:24:898.448 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:24:899.344 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:24:899.370 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692624899334 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 40063 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:24:899.378 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:24:899.640 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:24:899.727 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:24:899.771 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:24:899.785 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692624899761 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:24:899.792 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:25:553.066 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:25:553.134 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692625553029 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:25:553.160 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:25:553.420 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:25:553.460 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692625553393 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 38257 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:25:553.470 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:25:553.742 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:25:553.796 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:25:553.857 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:25:553.880 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692625553831 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:25:553.887 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:25:553.894 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:25:553.902 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692625553846 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:25:553.908 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:25:553.922 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:25:553.944 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692625553917 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:25:553.962 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:25:554.012 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:25:554.029 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692625554006 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:25:554.037 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:25:554.047 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:25:554.068 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692625554024 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:25:554.077 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:25:554.547 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:25:554.562 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692625554512 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:25:554.573 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:25:554.580 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:25:554.602 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692625554560 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:25:554.614 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:25:554.580 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:25:605.841 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:25:605.901 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692625605701 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:25:605.912 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:25:605.921 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:25:605.934 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692625605868 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 36971 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:25:605.940 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:25:606.207 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:25:606.321 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:25:606.383 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:25:606.410 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692625606372 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:25:606.419 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:26:185.055 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:26:185.106 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692626184902 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:26:185.116 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:26:185.124 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:26:185.153 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692626185075 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 33761 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:26:185.158 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:26:185.389 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:26:185.442 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:26:185.496 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:26:185.527 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692626185487 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:26:185.535 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:26:578.701 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:26:578.756 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692626578598 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:26:578.777 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:26:579.587 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:26:579.613 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692626579577 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 36331 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:26:579.620 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:26:579.943 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:26:580.147 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:26:580.197 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:26:580.217 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692626580177 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:26:580.225 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:27:926.283 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:27:926.349 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692627926155 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:27:926.363 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:27:926.799 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:27:926.834 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692627926772 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 41887 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:27:926.844 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:27:927.097 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:27:927.187 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:27:927.253 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:27:928.569 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692627927242 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:27:928.592 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:27:928.604 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:27:928.618 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692627927250 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:27:928.627 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:27:928.640 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:27:928.674 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692627927316 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:27:928.689 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:27:928.626 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:27:928.707 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:27:928.878 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692627927373 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:27:928.896 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:27:928.906 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:27:928.920 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692627928321 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:27:928.929 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:27:928.938 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:27:928.963 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692627928514 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:27:928.984 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:27:973.779 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:27:973.837 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692627973733 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:27:973.849 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:27:974.135 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:27:974.151 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692627974126 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 38001 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:27:974.157 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:27:974.481 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:27:974.750 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:27:974.790 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:27:974.803 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692627974783 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:27:974.809 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:28:536.037 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:28:536.215 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692628535962 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:28:536.226 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:28:536.235 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:28:536.247 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692628536125 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 45859 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:28:536.253 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:28:536.415 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:28:536.462 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:28:536.532 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:28:536.673 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692628536525 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:28:536.681 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:28:536.689 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:28:536.697 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692628536530 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:28:536.703 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:28:536.709 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:28:536.729 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692628536595 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:28:536.739 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:28:536.833 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:28:536.867 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692628536825 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:28:536.886 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:28:536.896 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:28:536.910 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692628536841 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:28:536.917 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:28:537.529 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:28:537.553 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692628537452 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:28:537.561 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:28:537.568 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:28:537.586 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692628537495 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:28:537.595 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:28:537.615 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:28:578.365 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:28:578.425 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692628578327 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:28:578.437 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:28:578.849 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:28:578.863 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692628578842 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 41967 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:28:578.870 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:28:579.231 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:28:579.299 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:28:579.335 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:28:579.349 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692628579327 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:28:579.356 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:28:587.675 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:28:587.739 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692628587628 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:28:587.751 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:28:588.056 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:28:588.103 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692628588047 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 41951 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:28:588.115 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:28:588.391 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:28:588.696 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:28:588.760 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:28:588.790 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692628588745 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:28:588.801 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:28:874.969 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:28:875.037 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692628874928 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:28:875.047 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:28:875.553 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:28:875.604 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692628875523 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 36187 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:28:875.617 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:28:875.831 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:28:875.909 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:28:876.010 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:28:876.037 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692628875949 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:28:876.045 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:29:624.008 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:29:624.083 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692629623980 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:29:624.093 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:29:624.412 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:29:624.436 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692629624406 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 43391 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:29:624.442 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:29:624.730 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:29:624.827 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:29:624.874 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:29:624.892 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692629624865 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:29:624.900 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:30:073.795 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:30:073.854 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692630073644 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:30:073.864 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:30:073.873 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:30:073.900 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692630073815 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 46789 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:30:073.906 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:30:074.122 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:30:074.169 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:30:074.224 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:30:074.234 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692630074217 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:30:074.240 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:30:074.256 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:30:074.264 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692630074221 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:30:074.269 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:30:074.309 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:30:074.331 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692630074303 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:30:074.342 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:30:074.379 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:30:074.409 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692630074372 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:30:074.420 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:30:074.969 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:30:075.022 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692630074910 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:30:075.034 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:30:075.046 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:30:075.076 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692630074956 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:30:075.089 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:30:075.123 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:30:125.348 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:30:125.412 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692630125243 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:30:125.425 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:30:125.787 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:30:125.829 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692630125738 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 45507 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:30:125.838 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:30:126.046 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:30:126.221 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:30:126.272 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:30:126.287 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692630126261 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:30:126.294 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:30:838.186 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:30:838.237 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692630838090 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:30:838.247 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:30:838.946 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:30:838.962 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692630838938 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 43959 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:30:839.000 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:30:839.408 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:30:839.603 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:30:839.648 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:30:839.674 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692630839632 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:30:839.684 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:31:062.354 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:31:062.409 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692631062326 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:31:062.712 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:31:062.721 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:31:062.732 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692631062492 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 46119 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:31:062.739 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:31:062.859 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:31:063.005 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:31:063.048 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:31:063.067 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692631063038 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:31:063.086 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:31:551.153 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:31:551.207 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692631551036 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:31:551.217 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:31:551.583 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:31:551.607 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692631551569 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 41817 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:31:551.614 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:31:551.952 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:31:552.054 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:31:552.114 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:31:552.135 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692631552090 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:31:552.142 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:32:251.456 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:32:251.527 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692632251429 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:32:251.543 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:32:251.677 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:32:251.697 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692632251669 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 41049 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:32:251.724 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:32:251.969 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:32:252.028 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:32:252.078 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:32:252.105 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692632252065 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:32:252.115 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:32:252.124 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:32:252.137 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692632252072 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:32:252.146 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:32:252.172 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:32:252.203 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692632252162 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:32:252.218 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:32:252.273 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:32:252.302 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692632252266 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:32:252.323 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:32:252.333 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:32:252.355 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692632252290 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:32:252.363 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:32:253.070 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:32:253.100 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692632253025 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:32:253.114 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:32:253.123 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:32:253.103 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:32:253.151 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692632253074 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:32:253.735 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:32:299.334 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:32:299.397 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692632299210 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:32:299.410 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:32:299.935 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:32:299.968 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692632299916 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 35865 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:32:299.998 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:32:300.223 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:32:300.294 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:32:300.360 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:32:300.408 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692632300350 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:32:300.417 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:32:899.319 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:32:899.369 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692632899162 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:32:899.378 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:32:900.441 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:32:900.470 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692632900430 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 41969 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:32:900.477 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:32:900.783 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:32:900.876 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:32:900.915 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:32:900.932 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692632900905 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:32:900.938 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:33:371.565 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:33:371.618 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692633371462 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:33:371.628 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:33:372.098 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:33:372.122 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692633372081 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 34547 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:33:372.141 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:33:372.377 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:33:372.445 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:33:372.584 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:33:372.598 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692633372516 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:33:372.605 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:34:511.868 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:34:511.922 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692634511767 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:34:511.932 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:34:512.671 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:34:512.696 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692634512645 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 44301 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:34:512.704 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:34:513.007 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:34:513.098 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:34:513.141 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:34:513.167 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692634513129 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:34:513.176 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:34:513.196 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:34:513.206 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692634513134 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:34:513.212 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:34:513.219 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:34:513.243 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692634513206 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:34:513.254 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:34:513.263 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:34:513.279 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692634513254 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:34:513.285 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:34:513.968 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:34:514.002 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692634513935 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:34:514.010 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:34:514.017 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:34:514.018 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:34:514.431 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692634513989 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:34:514.440 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:34:567.234 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:34:567.294 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692634567101 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:34:567.306 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:34:567.315 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:34:567.332 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692634567263 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 34483 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:34:567.341 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:34:567.565 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:34:567.630 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:34:567.696 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:34:567.717 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692634567678 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:34:567.727 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:35:280.763 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:35:280.814 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692635280643 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:35:280.824 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:35:281.257 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:35:281.285 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692635281241 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 43437 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:35:281.293 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:35:281.539 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:35:281.608 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:35:281.677 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:35:281.724 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692635281666 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:35:281.731 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:35:476.886 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:35:476.938 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692635476782 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:35:476.959 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:35:477.415 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:35:477.440 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692635477398 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 33669 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:35:477.447 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:35:477.752 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:35:477.816 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:35:477.876 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:35:477.907 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692635477867 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:35:477.913 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:35:764.360 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:35:764.410 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692635764215 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:35:764.420 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:35:764.903 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:35:764.931 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692635764881 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 39891 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:35:764.938 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:35:765.255 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:35:765.326 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:35:765.382 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:35:765.430 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692635765353 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:35:765.444 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:35:765.455 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:35:765.467 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692635765357 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:35:765.473 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:35:765.479 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:35:765.511 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692635765405 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:35:765.523 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:35:765.535 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:35:765.550 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692635765478 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:35:765.566 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:35:765.573 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:35:765.587 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692635765493 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:35:765.593 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:35:766.027 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:35:766.051 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692635765936 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:35:766.058 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:35:766.065 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:35:766.084 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692635765999 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:35:766.099 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:35:766.366 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:35:813.490 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:35:813.550 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692635813455 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:35:813.561 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:35:814.063 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:35:814.112 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692635814037 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 45699 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:35:814.121 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:35:814.383 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:35:814.434 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:35:814.480 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:35:814.518 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692635814472 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:35:814.525 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:36:573.084 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:36:573.156 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692636573044 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:36:573.185 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:36:573.209 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:36:573.226 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692636573201 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 40091 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:36:573.246 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:36:573.500 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:36:573.600 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:36:573.655 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:36:573.694 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692636573640 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:36:573.707 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:36:573.719 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:36:573.733 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692636573648 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:36:573.746 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:36:573.756 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:36:573.790 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692636573746 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:36:573.813 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:36:573.826 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:36:573.850 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692636573806 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:36:573.860 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:36:574.552 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:36:574.584 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692636574486 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:36:574.593 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:36:574.603 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:36:574.633 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692636574537 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:36:574.654 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:36:574.691 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:36:620.856 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:36:620.929 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692636620816 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:36:620.943 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:36:621.056 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:36:621.075 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692636621045 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 46353 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:36:621.082 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:36:621.443 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:36:621.553 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:36:621.604 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:36:621.621 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692636621592 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:36:621.627 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:36:858.671 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:36:858.723 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692636858567 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:36:858.734 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:36:859.151 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:36:859.176 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692636859132 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 37609 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:36:859.183 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:36:859.447 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:36:859.515 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:36:859.580 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:36:859.619 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692636859568 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:36:859.628 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:37:548.091 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:37:548.145 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692637547943 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:37:548.154 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:37:548.596 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:37:548.615 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692637548580 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 40439 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:37:548.628 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:37:548.903 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:37:548.984 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:37:549.059 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:37:549.098 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692637549049 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:37:549.105 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:37:566.749 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:37:566.805 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692637566713 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:37:566.815 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:37:567.085 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:37:567.100 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692637567078 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 40599 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:37:567.106 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:37:567.445 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:37:567.561 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:37:567.604 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:37:567.647 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692637567589 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:37:567.655 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:37:665.066 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:37:665.156 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692637664872 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 22 dequeues = 22 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 5990 dequeues = 5990 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 23 dequeues = 23 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 23 dequeues = 23 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 6060 dequeues = 6060 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756692608700694 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756692637567595 ] max_sm_queue_count = 7 ] 2025-09-01 Mon 02:10:37:665.183 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:10:37:665.192 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:37:665.310 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692637665019 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1195 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 107 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1359 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 327 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4612 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1317 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 3006, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 5990, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:10:37:665.325 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:10:37:670.861 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:37:670.912 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692637670830 [SYS_NOTICE]: UveVrouterAgent: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 xmpp_peer_list= [ [ [ ip = 10.20.0.129 status = 0 setup_time = 0 primary = 1 ], [ ip = 10.20.0.25 status = 0 setup_time = 0 primary = 0 ], ] ] dns_servers= [ [ (*_iter149) = 10.20.0.25, (*_iter149) = 10.20.0.193, ] ] ] 2025-09-01 Mon 02:10:37:670.922 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: UveVrouterAgent 2025-09-01 Mon 02:10:37:670.932 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:37:671.049 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692637670863 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 in_tpkts = 1712 in_bytes = 210945 out_tpkts = 1797 out_bytes = 100736 exception_packets = 61 exception_packets_dropped = 13 exception_packets_allowed = 48 raw_vhost_stats= [ name = vhost0 in_pkts = 1712 in_bytes = 210945 out_pkts = 1797 out_bytes = 100736 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 1 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 1 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 1 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 1 ] total_in_bandwidth_utilization = 6.9931e-15 total_out_bandwidth_utilization = 1.62793e-14 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens6 _iter260->second = 1290, ] ] phy_band_out_bps= [ [ _iter261->first = ens6 _iter261->second = 3003, ] ] raw_phy_if_stats= [ [ _iter262->first = ens6 [ in_pkts = 1623 in_bytes = 93428 out_pkts = 1714 out_bytes = 210997 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens6 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 02:10:37:671.070 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 02:10:37:671.079 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:37:671.106 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692637670894 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 raw_xmpp_stats= [ [ _iter306->first = 10.20.0.129 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], _iter306->first = 10.20.0.25 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 02:10:37:671.113 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 02:10:38:700.789 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 02:10:38:701.260 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Connect : Start Connect timer 10.0.0.249:8086 2025-09-01 Mon 02:10:38:701.354 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:10:38:701.382 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692638700893 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:38:701.395 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:38:701.403 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:10:38:701.513 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 02:10:38:701.541 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692638701341 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1195 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 109 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1359 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 327 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4612 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1317 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 3006, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 5995, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:10:38:701.578 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:10:38:701.550 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: -(-1):EvTcpConnectFail 2025-09-01 Mon 02:10:38:701.621 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 02:10:38:701.631 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Connect : EvTcpConnectFail 2025-09-01 Mon 02:10:38:701.726 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:38:701.759 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Idle 2025-09-01 Mon 02:10:38:701.797 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:38:701.918 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692638701681 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1196 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 110 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1359 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 327 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4612 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1317 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 3006, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 5997, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:10:38:701.935 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:10:38:701.943 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:38:701.959 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692638701706 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:38:701.983 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:38:701.990 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 02:10:38:702.016 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:38:702.133 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692638701789 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1196 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 110 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1360 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 327 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4612 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1317 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 3007, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 5997, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:10:38:702.150 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:10:38:767.796 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:38:767.861 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692638767719 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:38:767.873 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:38:768.150 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:38:768.201 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692638768122 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 38841 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:38:768.210 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:38:768.471 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:38:768.562 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:38:768.608 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:38:768.654 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692638768600 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:38:768.660 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:39:349.954 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:39:350.031 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692639349817 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:39:350.044 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:39:350.542 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:39:350.571 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692639350505 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 46201 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:39:350.583 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:39:350.815 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:39:350.924 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:39:350.988 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:39:351.017 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692639350960 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:39:351.025 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:39:351.033 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:39:351.041 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692639350978 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:39:351.047 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:39:351.053 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:39:351.074 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692639351040 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:39:351.083 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:39:351.147 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:39:351.162 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692639351142 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:39:351.173 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:39:351.180 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:39:351.193 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692639351159 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:39:351.199 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:39:351.796 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:39:351.812 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692639351761 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:39:351.820 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:39:351.828 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:39:351.854 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692639351823 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:39:351.865 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:39:352.288 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:39:401.420 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:39:401.490 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692639401385 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:39:401.501 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:39:401.868 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:39:401.883 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692639401860 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 46877 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:39:401.889 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:39:402.217 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:39:402.321 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:39:402.363 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:39:402.402 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692639402346 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:39:402.411 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:39:457.706 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:39:457.765 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692639457670 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:39:457.810 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:39:458.301 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:39:458.356 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692639458279 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 35871 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:39:458.365 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:39:459.030 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:39:459.123 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:39:459.178 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:39:459.202 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692639459167 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:39:459.209 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:40:365.547 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:40:365.598 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692640365450 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:40:365.608 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:40:365.896 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:40:365.920 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692640365883 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 34987 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:40:365.927 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:40:366.252 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:40:366.345 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:40:366.398 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:40:366.417 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692640366378 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:40:366.438 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:41:617.260 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:41:617.307 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692641617164 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:41:617.317 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:41:618.065 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:41:618.093 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692641618055 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 40491 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:41:618.100 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:41:618.365 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:41:618.419 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:41:618.504 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:41:618.520 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692641618446 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:41:618.526 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:41:618.533 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:41:618.541 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692641618450 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:41:618.546 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:41:619.143 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:41:619.188 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692641618588 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:41:619.213 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:41:619.221 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:41:619.236 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692641618629 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:41:619.243 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:41:619.249 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:41:619.258 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692641619083 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:41:619.265 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:41:619.271 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:41:619.284 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692641619126 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:41:619.295 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:41:619.318 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:41:665.498 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:41:665.550 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692641665405 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:41:665.561 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:41:666.132 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:41:666.159 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692641666122 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 38399 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:41:666.169 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:41:666.477 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:41:666.552 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:41:666.620 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:41:666.644 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692641666604 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:41:666.652 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:42:184.589 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:42:184.657 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692642184545 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:42:184.670 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:42:185.265 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:42:185.304 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692642185224 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 42315 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:42:185.313 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:42:185.555 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:42:185.648 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:42:185.689 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:42:185.705 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692642185678 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:42:185.713 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:42:683.836 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:42:683.902 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692642683735 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:42:683.915 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:42:684.431 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:42:684.480 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692642684397 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 37835 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:42:684.542 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:42:684.763 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:42:684.987 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:42:685.450 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:42:685.492 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692642685032 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:42:685.505 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:42:764.886 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:42:764.939 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692642764707 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:42:764.950 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:42:765.956 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:42:766.004 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692642765922 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 38623 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:42:766.017 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:42:766.284 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:42:767.377 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:42:767.470 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:42:767.485 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692642767416 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:42:767.493 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:42:767.512 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:42:767.521 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692642767427 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:42:767.527 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:42:768.238 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:42:768.272 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692642767567 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:42:768.283 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:42:768.292 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:42:768.307 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692642767647 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:42:768.317 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:42:768.343 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:42:768.365 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692642767662 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:42:768.374 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:42:768.383 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:42:768.396 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692642768151 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:42:768.403 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:42:768.409 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:42:768.423 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692642768195 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:42:768.430 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:42:768.449 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:42:820.474 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:42:820.564 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692642820393 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:42:820.573 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:42:820.983 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:42:821.024 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692642820946 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 45255 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:42:821.035 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:42:821.222 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:42:821.309 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:42:821.350 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:42:821.371 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692642821340 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:42:821.378 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:43:347.447 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:43:347.505 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692643347420 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:43:347.515 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:43:347.603 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:43:347.618 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692643347596 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 33387 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:43:347.625 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:43:347.948 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:43:348.076 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:43:348.197 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:43:348.226 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692643348106 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:43:348.237 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:43:348.251 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:43:348.261 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692643348158 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:43:348.267 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:43:348.273 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:43:348.298 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692643348222 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:43:348.313 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:43:348.324 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:43:348.346 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692643348288 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:43:348.362 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:43:348.834 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:43:348.849 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:43:348.856 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692643348800 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:43:348.918 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:43:348.926 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:43:348.947 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692643348830 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:43:348.957 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:43:394.037 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:43:394.194 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692643394007 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:43:394.216 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:43:394.224 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:43:394.234 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692643394164 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 41189 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:43:394.240 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:43:394.464 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:43:394.511 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:43:394.567 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:43:394.591 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692643394560 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:43:394.596 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:43:883.539 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:43:883.591 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692643883443 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:43:883.601 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:43:884.296 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:43:884.317 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692643884285 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 37715 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:43:884.324 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:43:884.556 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:43:884.606 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:43:884.694 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:43:884.706 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692643884639 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:43:884.713 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:44:465.724 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:44:465.787 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692644465630 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:44:465.803 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:44:466.440 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:44:466.456 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692644466429 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 43227 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:44:466.463 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:44:466.739 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:44:466.772 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:44:466.838 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:44:466.850 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692644466798 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:44:466.856 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:44:521.265 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:44:521.323 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692644521164 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:44:521.343 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:44:522.074 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:44:522.090 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692644522064 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 43045 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:44:522.096 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:44:522.381 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:44:522.418 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:44:522.486 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:44:522.498 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692644522444 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:44:522.504 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:45:918.807 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:45:918.866 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692645918767 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:45:918.877 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:45:919.365 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:45:919.404 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692645919339 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 33777 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:45:919.413 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:45:919.612 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:45:919.670 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:45:919.715 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:45:919.736 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692645919705 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:45:919.742 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:46:285.122 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:46:285.180 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692646284932 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:46:285.189 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:46:285.197 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:46:285.208 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692646285102 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 38825 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:46:285.214 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:46:285.414 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:46:285.473 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:46:285.522 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:46:285.542 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692646285499 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:46:285.550 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:46:470.040 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:46:470.108 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692646469797 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:46:470.137 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:46:470.150 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:46:470.167 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692646470004 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 43461 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:46:470.176 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:46:470.329 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:46:470.399 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:46:470.452 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:46:470.479 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692646470438 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:46:470.488 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:46:470.495 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:46:470.504 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692646470445 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:46:470.509 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:46:470.582 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:46:470.607 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692646470571 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:46:470.625 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:46:470.733 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:46:470.749 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692646470724 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:46:470.757 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:46:470.764 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:46:470.777 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692646470750 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:46:470.785 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:46:471.576 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:46:471.591 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692646471535 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:46:471.602 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:46:471.609 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:46:471.621 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:46:471.628 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692646471595 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:46:471.755 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:46:523.901 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:46:523.963 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692646523863 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:46:523.983 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:46:524.263 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:46:524.278 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692646524257 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 44351 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:46:524.284 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:46:524.594 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:46:524.698 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:46:524.739 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:46:524.763 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692646524729 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:46:524.770 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:47:496.986 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:47:497.041 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692647496875 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:47:497.052 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:47:497.840 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:47:497.868 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692647497822 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 37345 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:47:497.876 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:47:498.171 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:47:498.232 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:47:498.318 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:47:498.331 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692647498260 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:47:498.338 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:48:518.674 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:48:518.727 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692648518563 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:48:518.746 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:48:519.606 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:48:519.627 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692648519595 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 37127 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:48:519.634 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:48:519.874 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:48:519.907 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:48:519.999 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:48:520.012 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692648519933 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:48:520.019 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:48:520.025 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:48:520.033 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692648519936 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:48:520.039 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:48:520.566 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:48:520.622 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692648520077 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:48:520.646 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:48:520.651 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:48:520.656 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:48:520.757 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692648520117 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:48:520.768 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:48:520.780 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:48:520.795 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692648520588 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:48:520.803 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:48:520.813 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:48:520.834 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692648520627 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:48:520.853 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:48:566.727 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:48:566.783 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692648566701 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:48:566.793 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:48:567.054 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:48:567.065 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692648567047 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 39013 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:48:567.071 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:48:567.414 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:48:567.487 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:48:567.529 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:48:567.549 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692648567516 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:48:567.556 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:49:406.596 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:49:406.654 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692649406433 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:49:406.664 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:49:406.687 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:49:406.700 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692649406611 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 42773 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:49:406.706 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:49:406.892 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:49:406.943 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:49:407.012 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:49:407.036 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692649407003 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:49:407.043 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:49:494.701 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:49:494.771 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692649494670 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:49:495.510 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:49:496.217 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:49:496.250 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:49:496.272 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:49:496.273 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692649495209 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 35189 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:49:496.388 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:49:496.401 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:49:496.428 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692649496305 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:49:496.437 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:50:203.895 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:203.956 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692650203855 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:50:203.966 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:50:204.103 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:204.128 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692650204093 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 46165 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:50:204.134 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:50:204.394 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:50:204.465 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:50:204.693 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:204.861 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692650204663 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:50:204.873 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:50:204.882 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:204.891 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692650204674 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:50:204.897 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:50:204.916 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:204.939 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692650204745 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:50:204.950 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:50:205.021 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:205.065 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692650205014 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:50:205.076 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:50:205.084 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:205.098 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692650205031 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:50:205.111 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:50:205.798 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:205.818 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692650205707 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:50:205.827 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:50:205.837 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:205.855 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:50:205.874 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692650205807 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:50:205.884 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:50:259.055 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:259.164 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692650259024 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:50:259.176 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:50:259.186 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:259.211 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692650259180 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 43047 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:50:259.217 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:50:259.609 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:50:259.663 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:50:259.724 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:259.744 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692650259717 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:50:259.751 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:50:569.613 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:569.669 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692650569580 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:50:569.679 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:50:570.198 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:570.238 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692650570162 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 38961 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:50:570.247 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:50:570.461 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:50:570.509 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:50:570.548 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:570.572 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692650570540 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:50:570.579 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:50:570.594 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:570.602 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692650570546 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:50:570.607 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:50:570.613 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:570.636 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692650570607 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:50:570.646 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:50:570.680 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:570.700 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692650570672 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:50:570.711 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:50:571.388 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:571.411 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692650571343 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:50:571.418 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:50:571.427 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:571.446 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692650571383 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:50:571.453 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:50:571.455 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:50:623.684 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:623.736 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692650623654 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:50:623.746 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:50:623.895 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:623.906 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692650623890 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 37223 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:50:623.924 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:50:624.183 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:50:624.238 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:50:624.314 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:50:624.332 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692650624286 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:50:624.340 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:51:285.867 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:51:285.927 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692651285829 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:51:285.938 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:51:286.278 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:51:286.300 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692651286268 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 43463 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:51:286.321 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:51:286.749 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:51:286.991 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:51:287.030 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:51:287.045 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692651287021 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:51:287.064 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:51:404.562 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:51:404.618 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692651404533 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:51:404.629 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:51:404.978 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:51:405.002 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692651404960 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 38089 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:51:405.009 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:51:405.288 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:51:405.409 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:51:405.457 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:51:405.481 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692651405447 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:51:405.496 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:51:614.503 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:51:614.560 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692651614473 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:51:614.708 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:51:614.717 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:51:614.727 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692651614659 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 41193 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:51:614.747 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:51:614.928 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:51:614.997 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:51:615.057 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:51:615.087 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692651615049 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:51:615.094 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:53:249.376 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:53:249.443 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692653249207 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:53:249.454 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:53:249.464 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:53:249.476 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692653249435 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 42479 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:53:249.483 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:53:249.739 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:53:249.807 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:53:249.886 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:53:249.910 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692653249866 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:53:249.934 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:53:655.335 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:53:655.393 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692653655197 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:53:655.403 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:53:655.411 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:53:655.423 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692653655358 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 33543 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:53:655.430 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:53:655.627 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:53:655.675 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:53:655.731 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:53:655.752 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692653655723 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:53:655.758 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:53:655.765 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:53:655.774 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692653655728 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:53:655.779 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:53:655.814 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:53:655.855 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692653655808 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:53:655.959 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:53:655.976 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:53:655.992 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692653655892 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:53:656.000 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:53:656.008 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:53:656.022 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692653655911 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:53:656.037 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:53:656.468 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:53:656.519 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692653656421 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:53:656.529 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:53:656.536 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:53:656.556 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692653656455 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:53:656.569 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:53:656.562 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:53:701.808 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:53:701.872 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692653701682 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:53:701.885 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:53:702.815 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:53:702.846 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692653702798 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 36641 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:53:702.872 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:53:703.219 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:53:703.271 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:53:703.361 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:53:703.378 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692653703305 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:53:703.387 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:53:757.287 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:53:757.337 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692653757183 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:53:757.347 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:53:758.095 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:53:758.113 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692653758085 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 34207 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:53:758.120 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:53:758.355 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:53:758.392 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:53:758.458 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:53:758.470 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692653758417 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:53:758.488 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:54:684.582 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:54:685.882 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692654684441 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:54:685.909 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:54:685.464 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:54:685.922 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:54:685.981 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692654685133 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 41581 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:54:685.988 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:54:686.017 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:54:686.065 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:54:686.096 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692654686043 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:54:686.108 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:55:653.631 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:55:653.691 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692655653591 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:55:653.703 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:55:654.247 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:55:654.296 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692655654203 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 33701 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:55:654.324 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:55:654.512 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:55:654.561 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:55:654.601 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:55:654.614 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692655654592 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:55:654.621 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:55:654.628 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:55:654.637 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692655654596 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:55:654.643 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:55:654.686 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:55:654.712 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692655654679 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:55:654.724 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:55:654.733 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:55:654.749 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692655654720 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:55:654.763 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:55:655.309 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:55:655.333 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692655655260 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:55:655.340 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:55:655.347 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:55:655.368 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692655655297 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:55:655.377 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:55:655.399 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:55:701.544 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:55:701.843 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692655701513 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:55:701.858 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:55:701.866 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:55:701.877 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692655701667 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 34347 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:55:701.883 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:55:701.958 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:55:702.021 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:55:702.076 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:55:702.100 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692655702068 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:55:702.107 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:56:612.741 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:56:612.807 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692656612618 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:56:612.821 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:56:613.788 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:56:613.832 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692656613761 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 38197 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:56:613.844 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:56:614.044 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:56:614.114 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:56:614.389 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:56:614.415 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692656614324 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:56:614.425 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:56:719.305 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:56:719.361 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692656719198 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:56:719.371 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:56:720.088 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:56:720.108 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692656720078 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 41263 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:56:720.119 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:56:720.366 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:56:720.403 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:56:720.488 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:56:720.527 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692656720430 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:56:720.534 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:57:498.168 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:498.253 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692657498047 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:57:498.270 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:57:499.153 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:499.188 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692657499132 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 38343 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:57:499.198 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:57:499.471 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:57:499.524 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:57:500.545 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:500.578 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692657499556 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:57:500.589 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:57:500.599 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:500.613 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692657499564 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:57:500.621 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:57:500.631 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:500.635 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:57:500.665 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692657499665 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:57:500.691 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:57:500.703 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:500.729 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692657499788 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:57:500.740 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:57:500.750 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:500.773 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692657499810 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:57:500.786 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:57:500.796 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:500.808 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692657500438 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:57:500.817 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:57:500.826 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:500.850 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692657500476 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:57:500.861 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:57:545.818 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:546.252 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:57:546.301 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692657545785 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:57:546.302 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:57:546.313 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:57:546.407 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:546.423 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692657545942 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 35399 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:57:546.449 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:57:546.459 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:546.471 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692657546363 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:57:546.479 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:57:862.660 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:862.713 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692657862553 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:57:862.724 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:57:863.139 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:863.164 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692657863122 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 45057 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:57:863.171 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:57:863.403 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:57:863.468 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:57:863.528 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:863.539 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692657863518 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:57:864.356 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:57:864.203 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:10:57:864.368 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:864.425 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692657863527 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:57:864.432 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:57:864.440 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:864.462 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692657863582 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:57:864.472 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:57:864.486 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:864.535 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692657863624 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:10:57:864.542 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:57:864.548 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:864.561 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692657864129 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:57:864.567 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:57:864.573 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:864.586 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692657864186 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:10:57:864.594 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:10:57:909.311 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:909.366 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692657909276 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:57:909.376 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:57:909.605 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:909.621 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692657909596 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 36483 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:57:909.628 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:57:909.944 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:57:910.025 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:57:910.089 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:57:910.122 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692657910082 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:57:910.129 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:58:608.561 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:58:608.622 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692658608483 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:58:608.633 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:58:609.118 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:58:609.159 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692658609083 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 43381 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:58:609.169 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:58:609.369 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:58:609.434 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:58:609.470 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:58:609.485 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692658609462 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:58:609.528 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:58:772.684 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:58:772.739 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692658772585 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:58:772.749 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:58:773.448 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:58:773.468 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692658773436 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 34491 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:58:773.475 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:58:773.703 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:58:773.742 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:58:773.812 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:58:773.824 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692658773767 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:58:773.830 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:58:963.303 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:58:963.354 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692658963197 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:58:963.364 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:58:963.993 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:58:964.010 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692658963985 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 46023 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:58:964.016 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:10:58:964.607 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:10:58:964.703 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:10:58:964.782 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:10:58:964.794 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692658964740 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:10:58:964.800 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:00:679.732 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:00:679.787 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692660679625 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:00:679.798 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:00:680.147 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:00:680.174 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692660680133 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 37139 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:00:680.182 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:00:680.511 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:00:680.618 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:00:680.691 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:00:680.733 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692660680680 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:00:680.740 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:00:790.572 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:00:790.626 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692660790462 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:00:790.636 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:00:791.081 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:00:791.102 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692660791065 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 37177 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:00:791.109 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:00:791.401 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:00:791.475 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:00:791.541 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:00:791.553 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692660791530 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:00:791.560 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:00:791.566 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:00:791.575 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692660791539 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:00:791.580 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:00:791.634 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:00:791.667 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692660791624 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:00:791.700 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:00:791.720 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:00:791.736 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692660791705 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:00:791.744 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:00:791.751 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:00:791.765 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692660791730 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:00:791.771 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:00:792.304 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:00:792.326 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692660792252 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:00:792.333 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:00:792.347 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:00:792.367 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692660792289 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:00:792.377 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:00:792.411 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:00:844.630 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:00:844.688 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692660844597 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:00:844.699 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:00:845.239 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:00:845.281 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692660845209 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 40019 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:00:845.291 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:00:845.620 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:00:845.717 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:00:845.782 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:00:845.801 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692660845767 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:00:845.809 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:00:926.961 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:00:927.027 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692660926933 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:00:927.041 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:00:927.136 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:00:927.151 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692660927127 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 40865 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:00:927.168 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:00:927.524 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:00:927.590 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:00:927.641 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:00:927.676 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692660927618 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:00:927.686 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:01:907.953 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:01:908.162 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692661907921 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:01:908.173 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:01:908.196 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:01:908.208 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692661908085 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 40681 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:01:908.214 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:01:908.420 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:01:908.470 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:01:908.557 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:01:908.571 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692661908532 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:01:908.579 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:02:769.979 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:02:770.030 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692662769876 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:02:770.039 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:02:770.717 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:02:770.734 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692662770708 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 38817 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:02:770.741 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:02:771.133 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:02:771.342 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:02:771.391 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:02:771.422 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692662771370 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:02:771.430 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:02:771.437 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:02:771.446 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692662771379 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:02:771.451 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:02:771.471 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:02:771.492 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692662771428 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:02:771.502 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:02:771.513 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:02:771.528 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692662771487 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:02:771.536 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:02:771.988 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:02:772.012 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692662771948 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:02:772.022 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:02:772.034 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:02:772.063 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692662771995 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:02:772.075 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:02:772.246 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:02:821.515 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:02:821.578 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692662821406 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:02:821.588 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:02:821.945 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:02:821.961 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692662821930 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 41983 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:02:822.002 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:02:822.340 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:02:822.431 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:02:822.500 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:02:822.512 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692662822465 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:02:822.518 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:03:722.690 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:03:722.741 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692663722584 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:03:722.752 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:03:723.264 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:03:723.281 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692663723245 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 36229 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:03:723.288 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:03:723.549 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:03:723.642 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:03:723.717 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:03:723.759 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692663723706 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:03:723.783 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:03:888.665 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:03:888.727 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692663888630 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:03:888.741 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:03:889.167 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:03:889.189 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692663889155 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 45457 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:03:889.198 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:03:889.595 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:03:889.679 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:03:889.720 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:03:889.739 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692663889710 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:03:889.746 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:04:497.008 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:04:497.068 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692664496839 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:04:497.078 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:04:497.087 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:04:497.099 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692664497038 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 42139 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:04:497.119 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:04:497.400 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:04:497.457 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:04:497.519 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:04:497.638 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692664497511 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:04:497.647 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:04:497.655 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:04:497.664 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692664497518 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:04:497.670 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:04:497.676 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:04:497.713 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692664497571 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:04:497.723 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:04:497.787 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:04:497.839 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692664497779 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:04:497.853 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:04:497.866 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:04:497.888 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692664497796 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:04:497.897 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:04:498.890 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:04:498.893 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:04:498.932 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692664498309 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:04:499.081 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:04:499.106 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:04:499.129 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692664498852 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:04:499.139 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:04:551.046 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:04:551.104 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692664551005 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:04:551.116 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:04:551.484 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:04:551.504 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692664551475 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 35921 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:04:551.511 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:04:551.771 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:04:551.956 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:04:552.016 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:04:552.058 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692664551996 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:04:552.078 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:05:103.918 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:05:103.994 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692665103790 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:05:104.008 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:05:104.657 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:05:104.696 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692665104639 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 37371 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:05:104.707 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:05:105.018 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:05:105.122 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:05:105.175 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:05:105.206 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692665105162 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:05:105.215 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:05:105.224 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:05:105.233 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692665105170 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:05:105.238 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:05:105.302 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:05:105.326 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692665105295 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:05:105.349 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:05:105.358 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:05:105.373 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692665105351 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:05:105.379 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:05:106.080 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:05:106.111 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692665105998 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:05:106.121 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:05:106.131 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:05:106.161 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692665106053 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:05:106.183 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:05:106.204 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:05:158.334 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:05:158.386 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692665158219 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:05:158.397 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:05:159.295 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:05:159.338 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692665159281 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 37761 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:05:159.346 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:05:159.629 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:05:159.739 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:05:159.791 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:05:159.810 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692665159770 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:05:159.818 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:05:542.232 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:05:543.131 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:05:543.758 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692665542122 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:05:543.822 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:05:543.834 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:05:543.846 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692665542814 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 38961 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:05:543.852 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:05:543.868 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:05:543.915 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:05:543.928 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692665543902 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:05:543.935 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:05:595.933 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:05:596.043 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692665595828 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:05:596.053 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:05:596.453 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:05:596.512 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692665596406 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 44697 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:05:596.523 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:05:596.713 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:05:596.759 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:05:596.798 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:05:596.826 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692665596788 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:05:596.833 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:06:195.000 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:06:195.055 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692666194878 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:06:195.065 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:06:195.495 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:06:195.519 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692666195478 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 37921 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:06:195.526 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:06:195.842 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:06:195.905 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:06:195.952 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:06:195.992 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692666195933 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:06:196.003 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:07:542.142 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:07:542.206 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692667542045 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:07:542.217 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:07:542.923 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:07:542.949 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692667542913 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 37365 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:07:542.955 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:07:543.303 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:07:543.594 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:07:543.659 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:07:543.670 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692667543622 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:07:543.677 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:07:665.210 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:07:665.275 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692667665080 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 23 dequeues = 23 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 6283 dequeues = 6283 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 24 dequeues = 24 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 24 dequeues = 24 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 6356 dequeues = 6356 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756692638701758 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756692667543657 ] max_sm_queue_count = 7 ] 2025-09-01 Mon 02:11:07:665.287 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:11:07:665.307 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:07:665.434 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692667665171 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1253 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 112 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1425 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 343 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4839 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1382 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 3153, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6283, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:11:07:665.452 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:11:07:671.167 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:07:671.208 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692667671059 [SYS_NOTICE]: UveVrouterAgent: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 xmpp_peer_list= [ [ [ ip = 10.20.0.25 status = 0 setup_time = 0 primary = 1 ], [ ip = 10.20.0.193 status = 0 setup_time = 0 primary = 0 ], ] ] dns_servers= [ [ (*_iter149) = 10.20.0.129, (*_iter149) = 10.20.0.25, ] ] ] 2025-09-01 Mon 02:11:07:671.216 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: UveVrouterAgent 2025-09-01 Mon 02:11:07:671.223 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:07:671.249 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692667671108 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 2226148064 used = 32524 free = 0 buffers = 0 cached = 2303930040 node_type = ] meminfo= [ virt = 4294967295 peakvirt = 18275337 res = 0 ] cpuload= [ one_min_avg = 0.035 five_min_avg = 0.085 fifteen_min_avg = 0.115 ] cpu_share = 1.48333 ] ] 2025-09-01 Mon 02:11:07:671.256 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 02:11:07:671.263 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:07:671.275 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692667671125 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 cpu_info= [ [ [ mem_virt = 4294967295 cpu_share = 1.48333 used_sys_mem = 1991069780 one_min_cpuload = 0.035 mem_res = 0 ], ] ] ] 2025-09-01 Mon 02:11:07:671.287 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 02:11:07:671.294 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:07:671.384 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692667671148 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 in_tpkts = 1787 in_bytes = 216207 out_tpkts = 1874 out_bytes = 104818 exception_packets = 63 exception_packets_allowed = 50 raw_vhost_stats= [ name = vhost0 in_pkts = 1787 in_bytes = 216207 out_pkts = 1874 out_bytes = 104818 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 1 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 1 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 1 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 1 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens6 [ in_pkts = 1697 in_bytes = 97384 out_pkts = 1789 out_bytes = 216259 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens6 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 02:11:07:671.395 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 02:11:07:671.420 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:07:671.443 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692667671413 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 raw_xmpp_stats= [ [ _iter306->first = 10.20.0.193 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], _iter306->first = 10.20.0.25 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 02:11:07:671.449 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 02:11:08:032.151 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:08:032.223 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692668032108 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:08:032.235 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:08:032.677 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:08:032.701 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692668032654 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 39319 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:08:032.709 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:08:032.984 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:08:033.292 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:08:033.342 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:08:033.356 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692668033332 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:08:033.376 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:08:065.915 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:08:065.989 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692668065885 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:08:066.000 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:08:066.293 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:08:066.307 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692668066287 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 38477 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:08:066.314 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:08:066.643 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:08:066.749 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:08:066.793 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:08:066.821 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692668066781 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:08:066.829 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:08:066.847 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:08:066.856 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692668066788 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:08:066.861 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:08:066.870 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:08:066.904 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692668066862 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:08:066.914 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:08:067.280 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:08:067.297 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692668066939 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:08:067.305 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:08:067.315 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:08:067.329 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692668066954 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:08:067.339 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:08:067.457 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:08:067.470 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692668067447 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:08:067.477 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:08:067.493 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:08:067.509 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:08:067.512 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692668067486 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:08:067.662 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:08:119.667 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:08:119.726 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692668119629 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:08:119.737 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:08:120.056 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:08:120.071 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692668120047 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 38117 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:08:120.099 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:08:120.382 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:08:120.521 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:08:120.566 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:08:120.584 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692668120555 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:08:120.592 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:08:704.651 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 02:11:08:705.658 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Connect : Start Connect timer 10.0.0.38:8086 2025-09-01 Mon 02:11:08:705.787 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:11:08:705.815 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692668704736 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:08:705.827 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:08:705.840 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:11:08:706.006 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 02:11:08:706.043 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: -(-1):EvTcpConnectFail 2025-09-01 Mon 02:11:08:706.017 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692668705772 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1257 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 114 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1428 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 344 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4850 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1385 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 3160, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6305, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:11:08:706.161 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:11:08:706.176 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 02:11:08:706.183 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Connect : EvTcpConnectFail 2025-09-01 Mon 02:11:08:706.283 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:08:706.315 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Idle 2025-09-01 Mon 02:11:08:706.354 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:08:706.684 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692668706232 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.131:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1258 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 115 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1428 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 344 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4850 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1385 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 3160, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6307, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:11:08:706.709 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:11:08:706.719 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:08:706.737 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692668706262 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:08:706.745 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:08:706.758 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 02:11:08:706.786 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:08:706.932 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692668706345 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.131:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1258 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 115 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1429 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 344 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4850 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1385 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 3161, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6307, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:11:08:706.948 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:11:09:047.745 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:09:047.817 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692669047703 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:09:047.843 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:09:048.201 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:09:048.224 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692669048191 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 38953 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:09:048.232 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:09:048.480 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:09:048.568 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:09:048.625 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:09:048.646 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692669048618 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:09:048.653 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:09:593.003 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:09:593.055 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692669592866 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:09:593.066 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:09:593.453 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:09:593.485 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692669593435 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 44695 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:09:593.493 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:09:593.741 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:09:593.815 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:09:593.874 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:09:594.543 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:09:593.885 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692669593865 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:09:594.657 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:09:594.666 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:09:594.677 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692669593873 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:09:594.683 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:09:594.689 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:09:594.709 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692669593927 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:09:594.719 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:09:594.727 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:09:594.740 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692669593991 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:09:594.752 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:09:594.765 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:09:594.773 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692669594477 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:09:594.779 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:09:594.784 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:09:594.798 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692669594522 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:09:594.805 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:09:647.691 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:09:647.748 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692669647656 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:09:647.759 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:09:648.225 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:09:648.242 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692669648161 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 41963 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:09:648.249 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:09:648.553 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:09:648.661 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:09:648.698 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:09:648.714 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692669648691 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:09:648.720 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:10:593.935 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:10:595.304 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:10:595.441 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692670593780 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:10:595.487 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:10:595.496 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:10:595.508 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692670594885 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 40963 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:10:595.514 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:10:595.542 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:10:595.626 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:10:595.637 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692670595578 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:10:595.642 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:11:047.043 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:047.110 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692671046887 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:11:047.126 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:11:048.232 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:048.262 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692671048201 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 35165 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:11:048.271 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:11:048.558 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:11:048.630 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:11:048.701 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:048.714 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692671048664 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:11:048.721 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:11:633.488 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:633.540 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692671633454 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:11:633.550 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:11:634.048 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:634.095 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692671634022 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 38365 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:11:634.104 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:11:634.329 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:11:634.415 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:11:634.469 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:634.485 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692671634446 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:11:634.493 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:11:634.500 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:634.509 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692671634451 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:11:634.515 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:11:634.526 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:634.550 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692671634518 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:11:634.568 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:11:634.578 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:634.593 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692671634573 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:11:634.599 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:11:635.160 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:635.187 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692671635098 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:11:635.196 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:11:635.203 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:635.222 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692671635137 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:11:635.233 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:11:635.253 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:11:680.375 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:680.425 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692671680270 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:11:680.434 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:11:680.607 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:680.662 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692671680579 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 36555 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:11:680.671 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:11:680.912 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:11:680.960 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:11:681.007 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:681.020 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692671681001 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:11:681.026 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:11:755.810 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:755.864 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692671755781 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:11:755.874 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:11:756.154 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:756.170 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692671756148 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 44089 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:11:756.197 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:11:756.464 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:11:756.564 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:11:756.605 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:756.755 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692671756596 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:11:756.766 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:11:756.774 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:756.784 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692671756601 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:11:756.790 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:11:756.797 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:756.819 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692671756652 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:11:756.830 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:11:757.097 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:757.130 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692671757090 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:11:757.140 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:11:757.148 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:757.162 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692671757111 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:11:757.168 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:11:757.838 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:757.852 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692671757814 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:11:757.859 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:11:757.866 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:757.885 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692671757863 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:11:757.897 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:11:757.915 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:11:810.062 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:810.245 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692671810032 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:11:810.254 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:11:810.263 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:810.274 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692671810183 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 36343 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:11:810.280 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:11:810.510 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:11:810.557 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:11:810.681 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:11:810.704 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692671810648 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:11:810.729 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:12:671.258 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:12:671.313 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692672671143 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:12:671.327 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:12:671.630 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:12:671.653 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692672671611 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 39667 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:12:671.660 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:12:671.921 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:12:672.009 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:12:672.107 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:12:672.127 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692672672069 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:12:672.134 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:12:683.843 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:12:683.917 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692672683719 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:12:683.928 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:12:684.470 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:12:684.510 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692672684450 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 44131 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:12:684.534 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:12:684.790 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:12:684.826 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:12:684.898 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:12:684.910 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692672684855 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:12:684.916 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:12:710.839 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:12:710.896 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692672710729 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:12:710.906 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:12:711.672 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:12:711.695 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692672711657 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 42887 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:12:711.702 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:12:711.987 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:12:712.028 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:12:712.123 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:12:712.136 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692672712057 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:12:712.155 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:14:602.262 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:14:602.465 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692674602231 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:14:602.476 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:14:602.512 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:14:602.530 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692674602383 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 45779 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:14:602.539 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:14:602.708 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:14:602.751 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:14:602.868 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:14:602.879 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692674602802 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:14:602.885 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:14:616.241 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:14:616.429 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692674616213 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:14:616.439 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:14:616.447 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:14:616.457 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692674616364 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 34367 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:14:616.477 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:14:616.709 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:14:616.752 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:14:616.825 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:14:616.840 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692674616797 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:14:616.848 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:15:117.000 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:15:117.070 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692675116823 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:15:117.081 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:15:117.104 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:15:117.116 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692675117018 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 33113 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:15:117.122 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:15:117.362 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:15:117.408 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:15:117.463 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:15:117.483 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692675117455 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:15:117.504 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:15:117.511 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:15:117.519 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692675117461 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:15:117.524 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:15:117.548 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:15:117.571 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692675117544 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:15:117.582 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:15:117.625 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:15:117.643 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692675117621 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:15:117.650 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:15:117.661 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:15:117.675 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692675117637 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:15:117.681 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:15:118.185 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:15:118.211 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692675118140 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:15:118.219 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:15:118.226 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:15:118.265 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692675118172 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:15:118.275 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:15:118.315 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:15:167.475 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:15:167.550 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692675167426 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:15:167.565 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:15:167.909 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:15:167.976 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692675167896 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 44975 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:15:167.987 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:15:168.220 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:15:168.298 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:15:168.337 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:15:168.356 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692675168327 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:15:168.365 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:16:221.557 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:16:221.613 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692676221454 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:16:221.623 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:16:222.381 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:16:222.401 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692676222371 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 41787 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:16:222.420 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:16:222.692 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:16:222.729 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:16:222.814 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:16:222.825 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692676222754 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:16:222.832 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:16:357.092 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:16:357.146 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692676356992 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:16:357.155 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:16:357.919 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:16:357.944 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692676357909 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 34935 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:16:357.952 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:16:358.229 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:16:358.271 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:16:358.348 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:16:358.360 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692676358296 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:16:358.380 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:16:358.388 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:16:358.397 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692676358301 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:16:358.403 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:16:358.993 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:16:359.022 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:16:359.045 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692676358449 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:16:359.057 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:16:359.064 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:16:359.078 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692676358502 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:16:359.090 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:16:359.097 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:16:359.106 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692676358937 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:16:359.112 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:16:359.118 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:16:359.131 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692676358961 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:16:359.138 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:16:408.239 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:16:408.306 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692676408112 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:16:408.320 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:16:408.816 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:16:408.848 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692676408793 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 43901 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:16:408.858 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:16:409.117 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:16:409.398 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:16:409.451 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:16:409.468 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692676409438 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:16:409.475 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:17:453.675 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:17:453.729 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692677453575 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:17:453.740 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:17:454.499 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:17:454.534 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692677454481 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 43535 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:17:454.546 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:17:454.781 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:17:454.827 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:17:454.914 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:17:454.927 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692677454854 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:17:454.934 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:18:239.013 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:239.083 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692678238901 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:18:239.097 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:18:239.919 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:239.950 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692678239900 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 39241 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:18:239.961 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:18:240.206 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:18:240.245 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:18:240.319 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:240.333 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692678240272 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:18:240.340 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:18:815.069 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:815.120 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692678814954 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:18:815.130 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:18:815.942 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:815.960 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692678815933 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 35281 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:18:815.987 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:18:816.311 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:18:816.363 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:18:816.598 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:816.643 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692678816389 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:18:816.655 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:18:816.675 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:816.687 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692678816394 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:18:816.694 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:18:816.701 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:816.726 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692678816473 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:18:816.738 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:18:816.746 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:816.764 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692678816657 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:18:816.780 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:18:816.787 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:816.801 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692678816674 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:18:816.806 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:18:817.205 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:817.218 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692678817153 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:18:817.221 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:18:817.224 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:18:817.395 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:817.415 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692678817192 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:18:817.432 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:18:867.406 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:867.464 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692678867376 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:18:867.474 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:18:867.796 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:867.811 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692678867788 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 37193 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:18:867.817 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:18:868.096 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:18:868.156 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:18:868.200 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:868.239 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692678868183 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:18:868.248 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:18:901.007 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:901.077 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692678900945 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:18:901.103 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:18:901.168 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:901.188 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692678901158 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 35615 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:18:901.196 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:18:901.503 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:18:901.575 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:18:901.628 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:901.651 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692678901617 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:18:901.659 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:18:901.667 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:901.677 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692678901625 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:18:901.683 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:18:901.722 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:901.748 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692678901714 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:18:901.768 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:18:901.777 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:901.793 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692678901769 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:18:901.800 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:18:902.623 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:902.651 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692678902560 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:18:902.660 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:18:902.739 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:902.786 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692678902610 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:18:902.805 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:18:902.841 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:18:948.905 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:948.964 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692678948887 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:18:949.136 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:18:949.149 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:949.165 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692678949058 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 36923 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:18:949.174 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:18:949.451 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:18:949.698 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:18:949.744 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:18:949.769 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692678949731 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:18:949.779 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:19:399.271 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:19:399.327 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692679399224 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:19:399.336 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:19:399.717 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:19:399.736 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692679399708 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 42561 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:19:399.743 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:19:400.041 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:19:400.103 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:19:400.165 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:19:400.206 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692679400147 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:19:400.215 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:19:894.356 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:19:894.417 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692679894320 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:19:894.428 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:19:894.513 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:19:894.528 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692679894505 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 45709 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:19:894.535 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:19:894.805 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:19:894.866 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:19:894.923 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:19:894.945 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692679894911 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:19:894.953 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:19:984.903 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:19:984.961 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692679984858 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:19:984.985 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:19:985.566 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:19:985.605 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692679985533 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 39535 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:19:985.615 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:19:985.900 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:19:985.948 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:19:986.019 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:19:986.051 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692679986008 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:19:986.058 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:20:038.533 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.20.0.192:8085::10.20.0.193:40264 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 02:11:20:049.727 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.20.0.192:8085::10.20.0.193:40274 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 02:11:21:767.194 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:21:767.262 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692681767163 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:21:767.272 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:21:767.333 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:21:767.344 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692681767327 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 43107 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:21:767.358 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:21:767.614 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:21:767.669 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:21:767.726 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:21:767.746 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692681767717 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:21:767.753 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:21:858.336 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:21:858.391 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692681858187 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:21:858.409 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:21:858.418 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:21:858.430 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692681858363 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 35483 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:21:858.436 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:21:858.661 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:21:858.715 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:21:858.770 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:21:858.791 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692681858763 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:21:858.798 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:22:020.392 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:22:020.449 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692682020366 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:22:020.459 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:22:020.573 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:22:020.595 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692682020564 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 40401 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:22:020.605 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:22:020.847 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:22:020.909 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:22:020.951 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:22:020.982 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692682020942 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:22:020.990 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:22:021.009 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:22:021.018 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692682020948 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:22:021.024 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:22:021.040 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:22:021.060 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692682021032 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:22:021.070 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:22:021.135 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:22:021.156 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692682021128 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:22:021.164 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:22:021.172 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:22:021.189 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692682021154 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:22:021.195 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:22:021.661 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:22:021.683 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692682021619 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:22:021.690 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:22:021.697 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:22:021.722 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692682021649 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:22:021.732 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:22:021.761 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:22:067.009 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:22:067.090 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692682066880 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:22:067.105 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:22:067.451 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:22:067.479 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692682067430 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 41261 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:22:067.490 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:22:067.783 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:22:067.850 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:22:067.916 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:22:067.964 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692682067887 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:22:067.993 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:23:013.082 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:23:013.138 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692683012983 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:23:013.148 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:23:016.761 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:23:016.809 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692683016728 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 44437 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:23:016.822 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:23:017.078 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:23:017.172 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:23:017.235 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:23:017.274 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692683017226 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:23:017.281 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:23:504.330 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:23:504.389 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692683504291 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:23:504.398 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:23:504.869 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:23:504.911 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692683504844 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 39875 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:23:504.920 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:23:505.135 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:23:505.184 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:23:505.260 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:23:505.273 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692683505220 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:23:505.280 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:23:505.289 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:23:505.315 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692683505237 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:23:505.322 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:23:505.328 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:23:505.352 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692683505294 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:23:505.363 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:23:505.371 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:23:505.388 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692683505337 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:23:505.395 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:23:505.796 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:23:505.811 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692683505769 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:23:505.822 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:23:505.830 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:23:505.852 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692683505805 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:23:505.861 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:23:505.831 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:23:557.028 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:23:557.104 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692683556982 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:23:557.116 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:23:557.549 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:23:557.591 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692683557522 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 39129 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:23:557.600 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:23:557.938 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:23:558.427 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:23:558.524 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:23:558.553 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692683558475 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:23:558.561 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:24:602.847 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:24:602.914 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692684602681 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:24:602.925 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:24:602.934 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:24:602.946 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692684602866 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 36123 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:24:602.952 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:24:603.213 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:24:603.294 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:24:603.354 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:24:603.368 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692684603345 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:24:603.375 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:24:835.430 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:24:835.484 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692684835392 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:24:835.507 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:24:835.916 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:24:835.931 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692684835909 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 41709 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:24:835.946 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:24:836.288 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:24:836.524 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:24:836.577 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:24:836.598 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692684836564 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:24:836.605 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:25:495.124 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:495.185 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692685494962 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:25:495.197 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:25:495.209 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:495.221 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692685495142 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 35539 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:25:495.228 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:25:495.504 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:25:495.557 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:25:495.630 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:495.654 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692685495618 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:25:495.662 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:25:495.669 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:495.678 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692685495624 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:25:495.684 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:25:495.707 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:495.732 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692685495700 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:25:495.743 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:25:495.768 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:495.791 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692685495763 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:25:495.810 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:25:496.342 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:25:496.344 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:496.421 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692685496269 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:25:496.428 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:25:496.436 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:496.467 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692685496319 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:25:496.477 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:25:544.602 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:544.673 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692685544464 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:25:544.687 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:25:545.101 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:545.136 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692685545089 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 37929 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:25:545.146 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:25:545.403 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:25:545.727 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:25:545.769 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:545.792 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692685545754 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:25:545.802 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:25:655.872 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:655.931 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692685655831 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:25:655.942 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:25:656.335 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:656.348 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692685656328 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 38353 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:25:656.354 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:25:656.610 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:25:656.732 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:25:656.773 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:656.799 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692685656763 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:25:656.805 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:25:656.812 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:656.821 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692685656769 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:25:656.827 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:25:656.843 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:656.867 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692685656837 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:25:656.876 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:25:656.935 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:656.979 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692685656927 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:25:656.992 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:25:656.999 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:657.014 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692685656946 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:25:657.020 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:25:657.595 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:657.627 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692685657524 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:25:657.636 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:25:657.643 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:657.662 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692685657564 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:25:657.702 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:25:657.748 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:25:710.979 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:711.034 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692685710938 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:25:711.045 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:25:711.352 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:711.366 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692685711345 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 36019 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:25:711.381 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:25:711.642 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:25:711.734 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:25:711.789 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:25:711.806 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692685711768 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:25:711.814 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:26:464.141 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:26:464.202 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692686464103 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:26:464.214 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:26:464.584 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:26:464.647 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692686464558 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 37405 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:26:464.660 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:26:464.860 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:26:464.929 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:26:464.995 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:26:465.018 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692686464985 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:26:465.025 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:26:601.525 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:26:601.584 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692686601488 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:26:601.595 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:26:602.086 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:26:602.126 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692686602054 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 41713 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:26:602.135 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:26:602.373 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:26:602.447 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:26:602.577 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:26:602.638 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692686602561 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:26:602.661 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:26:756.163 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:26:756.219 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692686755962 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:26:756.230 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:26:756.239 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:26:756.251 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692686756203 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 42929 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:26:756.257 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:26:756.474 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:26:756.548 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:26:756.607 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:26:756.653 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692686756585 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:26:756.661 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:28:535.223 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:28:535.278 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692688535108 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:28:535.288 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:28:535.582 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:28:535.611 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692688535569 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 32831 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:28:535.618 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:28:535.867 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:28:535.951 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:28:536.015 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:28:536.062 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692688535988 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:28:536.072 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:28:544.776 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:28:544.830 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692688544681 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:28:544.840 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:28:545.173 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:28:545.190 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692688545158 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 40251 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:28:545.196 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:28:545.488 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:28:545.552 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:28:545.770 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:28:545.799 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692688545601 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:28:545.815 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:28:545.823 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:28:545.832 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692688545617 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:28:545.838 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:28:545.844 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:28:545.864 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692688545702 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:28:545.878 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:28:546.556 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:28:546.589 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692688545922 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:28:546.607 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:28:546.615 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:28:546.630 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692688545938 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:28:546.636 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:28:546.643 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:28:546.652 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692688546480 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:28:546.661 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:28:546.667 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:28:546.681 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692688546514 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:28:546.688 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:28:546.705 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:28:598.759 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:28:598.811 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692688598653 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:28:598.822 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:28:599.179 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:28:599.196 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692688599161 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 44399 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:28:599.203 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:28:599.433 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:28:599.509 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:28:599.593 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:28:599.606 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692688599561 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:28:599.615 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:28:862.834 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:28:862.888 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692688862721 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:28:862.898 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:28:863.439 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:28:863.503 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692688863395 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 43499 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:28:863.547 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:28:863.735 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:28:864.006 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:28:864.055 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:28:864.082 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692688864044 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:28:864.106 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:29:607.788 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:29:607.847 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692689607750 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:29:607.857 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:29:608.377 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:29:608.420 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692689608350 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 41373 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:29:608.429 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:29:608.644 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:29:608.700 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:29:608.746 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:29:608.773 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692689608734 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:29:608.793 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:30:454.806 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:30:454.868 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692690454702 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:30:454.882 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:30:455.646 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:30:455.668 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692690455633 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 41527 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:30:455.675 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:30:455.897 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:30:455.933 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:30:456.565 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:30:456.602 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692690455959 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:30:456.610 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:30:456.618 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:30:456.627 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692690455964 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:30:456.632 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:30:456.639 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:30:456.664 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:30:456.667 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692690456056 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:30:456.774 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:30:456.784 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:30:456.803 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692690456100 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:30:456.810 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:30:456.816 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:30:456.826 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692690456606 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:30:456.832 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:30:456.838 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:30:456.855 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692690456641 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:30:456.863 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:30:503.844 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:30:503.905 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692690503808 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:30:503.916 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:30:504.591 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:30:504.636 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692690504557 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 39783 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:30:504.648 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:30:504.927 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:30:504.994 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:30:505.049 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:30:505.090 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692690505036 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:30:505.100 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:31:414.283 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:31:414.336 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692691414175 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:31:414.346 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:31:414.642 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:31:414.659 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692691414629 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 42979 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:31:414.666 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:31:414.948 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:31:415.059 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:31:415.117 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:31:415.162 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692691415090 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:31:415.172 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:31:624.950 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:31:625.012 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692691624843 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:31:625.022 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:31:625.427 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:31:625.445 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692691625409 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 44977 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:31:625.465 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:31:625.691 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:31:625.775 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:31:625.846 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:31:625.888 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692691625831 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:31:625.910 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:32:496.287 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:32:496.410 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692692496165 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:32:496.425 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:32:496.999 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:32:497.025 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692692496963 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 41747 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:32:497.035 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:32:497.423 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:32:497.559 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:32:497.631 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:32:497.679 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692692497601 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:32:497.692 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:32:497.719 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:32:497.734 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692692497609 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:32:497.743 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:32:497.752 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:32:497.776 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692692497669 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:32:497.786 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:32:497.794 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:32:497.816 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692692497754 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:32:497.823 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:32:498.267 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:32:498.289 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692692498225 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:32:498.304 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:32:498.311 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:32:498.330 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692692498258 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:32:498.340 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:32:498.408 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:32:545.605 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:32:545.676 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692692545565 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:32:545.692 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:32:546.138 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:32:546.176 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692692546121 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 45179 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:32:546.186 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:32:546.391 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:32:546.513 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:32:546.555 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:32:546.656 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692692546540 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:32:546.669 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:33:112.277 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:33:112.353 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692693112238 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:33:112.364 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:33:112.688 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:33:112.713 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692693112679 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 33249 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:33:112.722 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:33:113.086 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:33:113.183 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:33:113.234 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:33:113.261 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692693113220 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:33:113.270 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:33:113.278 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:33:113.286 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692693113229 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:33:113.304 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:33:113.311 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:33:113.333 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692693113307 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:33:113.343 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:33:113.395 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:33:113.417 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692693113388 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:33:113.425 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:33:113.629 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:33:113.644 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692693113405 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:33:113.660 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:33:114.205 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:33:114.230 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692693114142 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:33:114.238 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:33:114.245 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:33:114.263 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692693114182 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:33:114.273 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:33:114.300 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:33:160.488 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:33:160.617 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692693160460 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:33:160.641 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:33:160.901 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:33:160.932 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692693160890 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 33497 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:33:160.942 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:33:161.243 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:33:161.353 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:33:161.397 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:33:161.418 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692693161386 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:33:161.426 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:33:377.331 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:33:377.404 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692693377296 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:33:377.417 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:33:377.930 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:33:377.990 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692693377904 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 44893 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:33:378.004 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:33:378.305 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:33:378.425 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:33:378.472 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:33:378.492 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692693378459 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:33:378.500 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:33:617.903 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:33:617.960 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692693617718 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:33:617.979 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:33:617.989 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:33:618.002 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692693617909 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 44045 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:33:618.008 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:33:618.198 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:33:618.296 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:33:618.397 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:33:618.414 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692693618357 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:33:618.420 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:34:106.564 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:34:106.626 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692694106524 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:34:106.649 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:34:107.219 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:34:107.256 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692694107138 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 40789 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:34:107.265 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:34:107.438 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:34:107.490 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:34:107.545 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:34:107.566 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692694107535 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:34:107.572 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:35:478.015 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:35:478.072 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692695477976 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:35:478.083 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:35:478.395 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:35:478.411 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692695478387 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 36659 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:35:478.418 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:35:478.727 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:35:478.832 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:35:478.913 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:35:478.930 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692695478866 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:35:478.938 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:35:478.956 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:35:478.972 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692695478872 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:35:478.978 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:35:478.989 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:35:479.012 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692695478946 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:35:479.451 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:35:479.466 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:35:479.484 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692695479047 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:35:479.497 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:35:479.503 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:35:479.517 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692695479062 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:35:479.523 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:35:479.530 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:35:479.539 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692695479525 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:35:479.544 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:35:479.561 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:35:479.573 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:35:479.576 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692695479553 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:35:479.795 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:35:490.575 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:35:490.628 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692695490547 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:35:490.638 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:35:490.985 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:35:491.006 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692695490977 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 34625 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:35:491.012 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:35:491.259 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:35:491.344 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:35:491.389 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:35:491.411 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692695491377 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:35:491.419 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:35:524.751 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:35:524.804 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692695524716 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:35:524.827 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:35:525.303 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:35:525.343 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692695525278 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 46255 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:35:525.353 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:35:525.550 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:35:525.598 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:35:525.641 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:35:525.663 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692695525632 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:35:525.670 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:36:195.789 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:36:195.851 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692696195759 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:36:195.865 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:36:196.441 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:36:196.483 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692696196407 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 45083 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:36:196.508 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:36:196.812 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:36:196.913 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:36:196.957 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:36:196.994 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692696196946 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:36:197.002 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:36:533.796 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:36:533.864 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692696533758 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:36:533.875 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:36:534.421 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:36:534.471 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692696534337 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 40559 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:36:534.483 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:36:534.653 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:36:534.710 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:36:534.782 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:36:534.802 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692696534747 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:36:534.813 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:37:554.689 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:37:555.514 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:37:556.233 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692697554574 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:37:556.284 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:37:556.296 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:37:556.309 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692697555175 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 39575 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:37:556.314 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:37:556.342 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:37:556.378 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:37:556.387 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692697556370 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:37:556.392 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:37:556.398 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:37:556.406 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692697556375 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:37:556.412 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:37:556.452 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:37:556.476 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692697556441 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:37:556.535 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:37:556.566 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:37:556.590 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692697556485 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:37:556.599 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:37:557.050 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:37:557.112 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692697557029 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:37:557.136 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:37:557.145 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:37:557.164 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692697557092 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:37:557.180 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:37:557.240 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:37:606.491 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:37:606.542 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692697606383 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:37:606.551 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:37:606.906 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:37:606.922 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692697606888 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 42395 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:37:606.929 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:37:607.178 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:37:607.257 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:37:607.378 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:37:607.414 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692697607311 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:37:607.423 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:37:665.430 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:37:665.504 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692697665298 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 24 dequeues = 24 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 6604 dequeues = 6604 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 25 dequeues = 25 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 25 dequeues = 25 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 6680 dequeues = 6680 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756692668706314 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756692697607370 ] max_sm_queue_count = 7 ] 2025-09-01 Mon 02:11:37:665.529 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:11:37:665.547 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:37:665.699 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692697665393 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.131:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1318 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 117 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1498 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 361 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5085 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1452 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 3314, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6604, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:11:37:665.718 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:11:37:671.321 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:37:671.384 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692697671286 [SYS_NOTICE]: UveVrouterAgent: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 xmpp_peer_list= [ [ [ ip = 10.20.0.193 status = 0 setup_time = 0 primary = 1 ], [ ip = 10.20.0.25 status = 0 setup_time = 0 primary = 0 ], ] ] dns_servers= [ [ (*_iter149) = 10.20.0.25, (*_iter149) = 10.20.0.129, ] ] ] 2025-09-01 Mon 02:11:37:671.398 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: UveVrouterAgent 2025-09-01 Mon 02:11:37:671.412 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:37:671.642 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692697671321 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 in_tpkts = 1874 in_bytes = 232740 out_tpkts = 1965 out_bytes = 110266 exception_packets = 64 exception_packets_allowed = 51 raw_vhost_stats= [ name = vhost0 in_pkts = 1874 in_bytes = 232740 out_pkts = 1965 out_bytes = 110266 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 1 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 1 ] phy_if_5min_usage= [ [ [ name = ens6 in_bandwidth_usage = 1259 out_bandwidth_usage = 2949 ], ] ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 1 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 1 ] total_in_bandwidth_utilization = 6.70579e-15 total_out_bandwidth_utilization = 1.57535e-14 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens6 _iter260->second = 1237, ] ] phy_band_out_bps= [ [ _iter261->first = ens6 _iter261->second = 2906, ] ] raw_phy_if_stats= [ [ _iter262->first = ens6 [ in_pkts = 1785 in_bytes = 102706 out_pkts = 1876 out_bytes = 232792 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens6 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 02:11:37:671.669 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 02:11:37:671.680 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:37:671.709 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692697671346 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 raw_xmpp_stats= [ [ _iter306->first = 10.20.0.193 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], _iter306->first = 10.20.0.25 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 02:11:37:671.715 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 02:11:38:624.523 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:38:624.756 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692698624478 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:38:624.766 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:38:624.775 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:38:624.786 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692698624660 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 43531 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:38:624.792 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:38:624.988 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:38:625.078 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:38:625.155 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:38:625.166 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692698625128 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:38:625.172 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:38:658.933 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:38:659.014 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692698658896 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:38:659.025 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:38:659.466 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:38:659.495 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692698659441 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 45911 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:38:659.504 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:38:659.725 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:38:660.173 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:38:660.230 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:38:660.245 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692698660202 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:38:660.252 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:38:706.401 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 02:11:38:707.041 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Connect : Start Connect timer 10.0.0.131:8086 2025-09-01 Mon 02:11:38:707.154 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:11:38:707.181 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692698706483 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.131:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:38:707.212 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:38:707.221 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:11:38:707.269 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 02:11:38:707.277 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: -(-1):EvTcpConnectFail 2025-09-01 Mon 02:11:38:707.402 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692698707136 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.131:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1318 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 119 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1500 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 361 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5091 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1454 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 3318, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6615, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:11:38:707.420 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:11:38:707.428 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 02:11:38:707.434 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Connect : EvTcpConnectFail 2025-09-01 Mon 02:11:38:707.514 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:38:707.544 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Idle 2025-09-01 Mon 02:11:38:707.583 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:38:707.691 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692698707472 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1319 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 120 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1500 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 361 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5091 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1454 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 3318, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6617, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:11:38:707.713 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:11:38:707.722 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:38:707.737 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692698707497 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:38:707.745 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:38:707.752 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 02:11:38:707.777 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:38:707.880 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692698707575 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1319 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 120 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1501 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 361 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5091 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1454 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 3319, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6617, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:11:38:707.896 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:11:39:487.851 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:39:487.930 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692699487636 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:39:487.949 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:39:487.963 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:39:488.004 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692699487825 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 36297 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:39:488.013 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:39:488.168 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:39:488.237 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:39:488.302 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:39:488.331 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692699488290 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:39:488.342 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:39:488.353 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:39:488.366 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692699488296 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:39:488.374 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:39:488.403 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:39:488.453 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692699488393 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:39:488.468 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:39:488.479 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:39:488.519 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692699488460 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:39:488.538 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:39:489.122 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:39:489.144 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:39:489.147 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692699489071 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:39:489.211 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:39:489.229 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:39:489.251 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692699489122 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:39:489.261 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:39:537.268 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:39:537.326 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692699537235 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:39:537.336 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:39:537.934 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:39:537.997 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692699537898 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 44569 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:39:538.012 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:39:538.217 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:39:538.272 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:39:538.343 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:39:538.365 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692699538312 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:39:538.389 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:40:121.155 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:40:121.214 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692700121117 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:40:121.224 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:40:121.605 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:40:121.618 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692700121596 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 43383 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:40:121.624 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:40:121.918 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:40:122.015 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:40:122.058 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:40:122.083 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692700122047 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:40:122.091 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:40:122.100 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:40:122.113 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692700122053 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:40:122.121 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:40:122.188 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:40:122.241 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692700122177 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:40:122.314 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:40:122.324 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:40:122.340 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692700122268 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:40:122.347 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:40:122.354 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:40:122.369 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692700122283 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:40:122.375 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:40:122.752 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:40:122.765 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692700122745 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:40:122.771 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:40:122.777 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:40:122.795 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692700122767 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:40:122.803 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:40:122.818 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:40:168.963 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:40:169.030 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692700168935 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:40:169.040 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:40:169.377 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:40:169.406 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692700169365 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 35015 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:40:169.454 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:40:169.672 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:40:169.761 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:40:169.803 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:40:169.828 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692700169793 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:40:169.836 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:40:569.473 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:40:569.532 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692700569304 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:40:569.543 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:40:569.551 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:40:569.562 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692700569481 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 38451 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:40:569.567 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:40:569.795 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:40:569.855 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:40:569.923 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:40:569.958 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692700569914 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:40:569.964 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:40:582.489 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:40:582.561 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692700582448 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:40:582.578 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:40:583.053 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:40:583.090 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692700583030 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 40603 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:40:583.101 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:40:583.403 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:40:583.467 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:40:583.529 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:40:583.544 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692700583500 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:40:583.551 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:41:132.978 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:41:133.036 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692701132936 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:41:133.047 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:41:133.380 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:41:133.408 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692701133369 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 33951 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:41:133.427 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:41:133.716 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:41:133.808 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:41:133.853 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:41:133.876 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692701133842 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:41:133.884 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:42:581.898 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:42:582.740 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:42:583.489 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692702581786 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:42:583.514 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:42:583.525 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:42:583.538 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692702582387 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 40447 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:42:583.544 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:42:583.598 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:42:583.637 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:42:583.652 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692702583627 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:42:583.660 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:42:656.407 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:42:656.457 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692702656302 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:42:656.467 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:42:656.895 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:42:656.911 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692702656883 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 38225 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:42:656.918 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:42:657.165 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:42:657.231 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:42:657.292 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:42:657.303 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692702657283 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:42:657.466 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:42:657.474 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:42:657.484 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692702657292 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:42:657.490 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:42:657.496 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:42:657.517 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692702657348 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:42:657.536 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:42:658.192 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:42:658.223 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:42:658.253 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692702657611 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:42:658.263 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:42:658.271 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:42:658.285 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692702657626 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:42:658.296 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:42:658.303 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:42:658.312 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692702658110 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:42:658.318 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:42:658.324 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:42:658.337 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692702658136 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:42:658.346 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:42:708.461 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:42:708.712 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692702708300 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:42:708.724 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:42:708.949 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:42:708.982 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692702708927 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 44927 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:42:708.990 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:42:709.402 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:42:709.514 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:42:709.570 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:42:709.590 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692702709555 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:42:709.598 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:43:132.172 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:43:132.229 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692703132012 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:43:132.240 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:43:132.249 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:43:132.261 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692703132191 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 44899 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:43:132.267 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:43:132.517 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:43:132.569 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:43:132.639 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:43:132.656 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692703132631 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:43:132.662 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:43:780.765 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:43:781.360 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:43:781.506 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:43:781.780 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692703780724 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:43:781.792 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:43:781.846 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:43:781.859 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692703780897 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 44483 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:43:781.865 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:43:781.871 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:43:781.879 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692703781541 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:43:781.884 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:44:170.208 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:44:170.264 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692704170179 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:44:170.274 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:44:170.361 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:44:170.382 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692704170352 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 45365 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:44:170.407 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:44:170.786 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:44:170.860 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:44:170.901 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:44:170.924 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692704170890 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:44:170.931 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:44:170.939 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:44:170.948 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692704170897 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:44:170.953 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:44:170.986 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:44:171.008 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692704170980 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:44:171.018 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:44:171.026 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:44:171.041 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692704171021 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:44:171.054 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:44:171.509 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:44:171.530 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692704171452 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:44:171.537 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:44:171.544 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:44:171.554 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:44:171.563 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692704171488 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:44:171.722 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:44:223.809 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:44:224.013 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692704223778 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:44:224.045 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:44:224.392 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:44:224.420 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692704224380 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 45093 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:44:224.430 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:44:224.724 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:44:224.841 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:44:224.910 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:44:224.956 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692704224888 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:44:224.977 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:45:161.071 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:45:161.128 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692705161035 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:45:161.139 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:45:161.667 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:45:161.680 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692705161637 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 43943 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:45:161.686 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:45:161.962 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:45:162.059 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:45:162.124 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:45:162.151 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692705162091 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:45:162.162 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:45:635.887 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:45:635.985 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692705635721 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:45:635.999 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:45:637.266 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:45:637.314 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692705637211 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 37689 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:45:637.326 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:45:638.494 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:45:638.912 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:45:639.123 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:45:639.148 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692705638998 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:45:639.185 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:46:733.009 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:733.062 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692706732782 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:46:733.082 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:46:733.394 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:733.416 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692706733378 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 43583 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:46:733.509 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:46:733.727 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:46:733.828 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:46:733.893 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:734.067 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692706733882 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:46:734.084 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:46:734.094 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:734.107 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692706733890 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:46:734.115 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:46:734.123 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:734.153 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692706733974 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:46:734.170 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:46:734.788 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:734.830 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692706734231 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:46:734.841 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:46:734.848 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:734.862 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692706734247 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:46:734.869 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:46:734.876 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:734.885 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692706734725 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:46:734.896 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:46:734.902 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:734.916 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692706734749 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:46:734.923 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:46:734.945 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:46:787.979 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:788.036 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692706787934 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:46:788.046 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:46:788.545 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:788.592 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692706788519 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 37151 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:46:788.602 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:46:788.855 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:46:788.914 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:46:788.978 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:788.994 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692706788947 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:46:789.002 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:46:832.009 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:832.061 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692706831888 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:46:832.071 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:46:832.591 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:832.608 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692706832571 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 35293 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:46:832.614 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:46:832.953 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:46:833.030 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:46:833.083 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:833.108 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692706833058 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:46:833.119 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:46:833.128 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:833.136 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692706833062 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:46:833.155 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:46:833.165 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:833.187 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692706833126 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:46:833.200 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:46:833.307 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:833.324 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692706833166 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:46:833.332 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:46:833.690 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:833.711 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692706833609 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:46:833.718 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:46:833.732 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:833.750 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692706833681 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:46:833.760 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:46:833.800 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:46:883.064 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:883.126 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692706882932 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:46:883.140 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:46:883.586 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:883.605 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692706883569 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 43461 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:46:883.619 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:46:883.947 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:46:884.089 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:46:884.210 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:46:884.242 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692706884194 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:46:884.251 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:47:214.383 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:47:214.456 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692707214236 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:47:214.471 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:47:214.891 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:47:214.924 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692707214873 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 43051 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:47:214.932 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:47:215.199 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:47:215.267 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:47:215.329 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:47:215.373 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692707215321 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:47:215.379 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:47:779.249 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:47:779.329 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692707779115 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:47:779.340 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:47:779.365 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:47:779.378 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692707779271 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 34451 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:47:779.384 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:47:779.539 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:47:779.592 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:47:779.650 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:47:779.673 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692707779642 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:47:779.680 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:47:928.361 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:47:928.419 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692707928323 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:47:928.429 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:47:928.653 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:47:928.674 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692707928642 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 34569 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:47:928.681 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:47:928.943 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:47:929.183 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:47:929.240 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:47:929.269 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692707929227 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:47:929.277 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:49:383.207 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:49:383.281 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692709383153 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:49:383.293 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:49:383.858 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:49:383.899 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692709383818 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 40187 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:49:383.908 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:49:384.147 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:49:384.244 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:49:384.286 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:49:384.303 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692709384276 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:49:384.313 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:49:384.324 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:49:384.337 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692709384280 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:49:384.343 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:49:384.364 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:49:384.387 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692709384335 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:49:384.397 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:49:384.433 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:49:384.455 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692709384427 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:49:384.465 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:49:384.475 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:49:384.510 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692709384444 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:49:384.522 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:49:385.214 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:49:385.242 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692709385151 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:49:385.250 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:49:385.257 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:49:385.375 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:49:385.653 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692709385198 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:49:385.664 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:49:430.548 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:49:430.602 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692709430520 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:49:430.612 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:49:431.000 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:49:431.021 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692709430990 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 38675 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:49:431.027 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:49:431.318 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:49:431.515 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:49:431.562 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:49:431.597 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692709431542 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:49:431.608 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:49:813.918 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:49:813.986 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692709813785 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:49:813.997 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:49:814.562 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:49:814.585 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692709814534 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 37027 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:49:814.593 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:49:814.840 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:49:814.912 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:49:814.987 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:49:815.045 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692709814964 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:49:815.052 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:50:071.566 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:50:071.631 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692710071384 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:50:071.645 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:50:072.090 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:50:072.156 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692710072061 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 34741 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:50:072.165 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:50:072.369 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:50:072.441 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:50:072.550 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:50:072.572 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692710072539 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:50:072.578 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:50:448.638 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:50:448.692 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692710448608 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:50:448.703 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:50:448.855 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:50:448.915 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692710448835 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 46441 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:50:448.925 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:50:449.104 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:50:449.148 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:50:449.188 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:50:449.226 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692710449178 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:50:449.233 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:51:391.552 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:51:391.604 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692711391442 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:51:391.614 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:51:392.035 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:51:392.061 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692711392021 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 40441 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:51:392.069 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:51:392.325 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:51:392.386 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:51:392.428 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:51:392.466 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692711392416 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:51:392.474 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:51:392.482 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:51:392.502 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692711392420 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:51:392.509 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:51:392.750 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:51:392.775 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692711392724 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:51:392.785 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:51:392.794 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:51:392.808 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692711392780 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:51:392.818 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:51:393.250 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:51:393.264 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692711393219 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:51:393.270 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:51:393.280 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:51:393.298 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692711393252 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:51:393.309 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:51:393.271 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:51:438.450 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:51:438.502 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692711438420 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:51:438.511 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:51:438.591 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:51:438.614 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692711438584 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 44767 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:51:438.632 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:51:438.861 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:51:438.922 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:51:438.957 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:51:438.992 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692711438948 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:51:439.000 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:52:366.255 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:52:366.324 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692712366115 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:52:366.338 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:52:366.349 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:52:366.360 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692712366273 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 41931 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:52:366.366 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:52:366.618 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:52:366.670 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:52:366.755 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:52:366.769 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692712366719 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:52:366.787 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:52:429.368 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:52:429.422 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692712429339 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:52:429.433 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:52:429.697 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:52:429.711 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692712429690 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 34779 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:52:429.729 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:52:429.985 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:52:430.054 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:52:430.088 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:52:430.099 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692712430081 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:52:430.105 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:53:703.179 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:53:703.473 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692713703149 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:53:703.612 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:53:703.647 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:53:703.678 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:53:703.683 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:53:704.085 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692713703310 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 42045 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:53:704.097 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:53:704.110 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:53:704.120 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692713703747 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:53:704.125 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:53:704.132 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:53:704.140 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692713703753 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:53:704.146 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:53:704.152 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:53:704.172 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692713703843 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:53:704.182 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:53:704.225 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:53:704.242 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692713703947 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:53:704.256 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:53:704.263 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:53:704.276 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692713703963 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:53:704.282 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:53:704.521 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:53:704.544 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692713704439 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:53:704.552 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:53:704.562 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:53:704.581 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692713704471 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:53:704.597 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:53:704.553 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:53:757.678 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:53:757.736 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692713757642 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:53:757.749 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:53:758.232 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:53:758.251 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692713758223 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 33073 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:53:758.266 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:53:758.570 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:53:758.827 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:53:758.865 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:53:758.885 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692713758853 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:53:758.895 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:54:140.786 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:54:140.841 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692714140669 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:54:140.854 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:54:141.312 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:54:141.338 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692714141296 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 44491 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:54:141.345 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:54:141.642 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:54:141.720 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:54:141.793 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:54:142.464 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:54:142.559 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692714141780 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:54:142.573 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:54:142.587 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:54:142.603 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692714141792 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:54:142.612 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:54:142.621 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:54:142.664 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692714141850 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:54:142.678 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:54:142.689 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:54:142.711 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692714141894 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:54:142.720 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:54:142.738 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:54:142.751 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692714142365 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:54:142.759 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:54:142.769 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:54:142.784 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692714142447 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:54:142.796 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:54:194.583 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:54:194.640 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692714194548 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:54:194.658 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:54:195.016 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:54:195.032 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692714195006 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 39671 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:54:195.039 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:54:195.336 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:54:195.438 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:54:195.491 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:54:195.532 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692714195472 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:54:195.540 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:54:490.867 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:54:490.929 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692714490834 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:54:490.954 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:54:491.430 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:54:491.446 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692714491400 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 44277 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:54:491.453 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:54:491.712 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:54:491.773 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:54:491.809 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:54:491.845 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692714491798 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:54:491.855 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:54:749.091 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:54:749.151 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692714748975 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:54:749.164 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:54:749.474 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:54:749.502 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692714749457 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 33067 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:54:749.509 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:54:749.839 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:54:750.084 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:54:750.149 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:54:750.180 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692714750124 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:54:750.189 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:55:158.639 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:55:158.691 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692715158546 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:55:158.701 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:55:159.357 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:55:159.385 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692715159345 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 37697 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:55:159.393 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:55:159.648 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:55:159.705 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:55:159.769 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:55:159.781 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692715159734 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:55:159.787 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:56:570.298 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:56:570.347 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692716570199 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:56:570.367 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:56:571.070 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:56:571.094 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692716571059 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 33777 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:56:571.102 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:56:571.395 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:56:571.451 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:56:571.514 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:56:571.525 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692716571477 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:56:571.530 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:56:571.537 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:56:571.544 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692716571480 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:56:571.550 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:56:571.675 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:56:571.710 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692716571588 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:56:571.730 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:56:571.745 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:56:571.762 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692716571665 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:56:571.769 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:56:571.809 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:56:571.822 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692716571680 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:56:571.829 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:56:572.322 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:56:572.358 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:56:572.369 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692716572175 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:56:572.375 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:56:572.382 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:56:572.401 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692716572282 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:56:572.411 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:56:625.513 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:56:625.672 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692716625477 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:56:625.683 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:56:625.691 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:56:625.703 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692716625640 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 36351 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:56:625.709 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:56:625.940 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:56:626.016 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:56:626.086 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:56:626.118 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692716626076 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:56:626.125 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:56:712.430 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:56:712.486 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692716712292 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:56:712.509 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:56:712.518 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:56:712.531 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692716712449 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 39807 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:56:712.538 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:56:712.821 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:56:712.868 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:56:712.935 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:56:712.953 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692716712917 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:56:712.959 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:57:175.978 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:57:176.037 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692717175934 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:57:176.062 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:57:176.139 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:57:176.171 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692717176129 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 37243 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:57:176.181 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:57:176.445 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:57:176.549 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:57:176.599 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:57:176.616 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692717176583 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:57:176.623 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:57:634.317 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:57:634.368 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692717634213 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:57:634.378 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:57:635.153 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:57:635.178 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692717635144 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 39997 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:57:635.185 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:57:635.520 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:57:635.623 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:57:635.678 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:57:635.701 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692717635655 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:57:635.712 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:58:128.035 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:58:128.089 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692718127911 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:58:128.100 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:58:128.549 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:58:128.570 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692718128488 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 43889 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:58:128.577 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:58:128.834 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:58:128.906 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:58:128.975 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:58:129.633 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:11:58:129.740 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692718128956 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:58:129.759 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:58:129.834 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:58:129.845 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692718128973 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:58:129.850 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:58:129.861 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:58:129.883 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692718129025 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:58:129.893 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:58:129.900 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:58:129.914 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692718129075 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:11:58:129.920 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:58:129.926 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:58:129.937 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692718129564 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:58:129.943 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:58:129.949 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:58:129.962 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692718129611 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:11:58:129.978 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:11:58:175.844 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:58:175.904 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692718175740 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:58:175.914 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:58:176.317 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:58:176.340 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692718176299 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 43043 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:58:176.347 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:58:176.597 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:58:176.666 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:58:176.753 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:58:176.786 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692718176744 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:58:176.793 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:59:122.012 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:59:122.079 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692719121869 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:59:122.093 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:59:122.928 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:59:122.951 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692719122913 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 43385 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:59:122.959 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:59:123.307 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:59:127.548 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:59:127.663 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:59:127.681 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692719127606 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:59:127.688 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:59:579.827 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:59:579.901 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692719579788 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:59:579.911 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:59:580.484 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:59:580.540 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692719580464 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 38919 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:59:580.548 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:11:59:580.762 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:11:59:580.818 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:11:59:580.864 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:11:59:580.878 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692719580855 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:11:59:580.884 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:00:313.164 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:313.223 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692720313135 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:00:313.232 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:00:313.363 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:313.388 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692720313353 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 36949 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:00:313.395 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:00:313.761 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:00:313.833 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:00:313.890 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:313.920 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692720313879 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:00:313.930 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:00:313.941 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:313.955 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692720313884 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:00:313.977 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:00:313.987 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:314.050 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692720313955 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:00:314.064 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:00:314.076 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:314.101 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692720314048 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:00:314.119 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:00:314.130 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:314.152 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692720314064 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:12:00:314.161 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:00:314.582 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:314.612 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692720314546 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:00:314.621 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:00:314.631 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:314.643 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:12:00:314.662 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692720314584 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:00:315.079 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:00:365.849 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:365.907 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692720365731 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:00:365.917 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:00:366.406 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:366.428 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692720366384 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 37993 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:00:366.438 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:00:366.700 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:00:366.770 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:00:366.832 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:366.868 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692720366821 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:00:366.877 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:00:884.799 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:884.861 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692720884761 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:00:884.999 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:00:885.022 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:885.034 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692720884959 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 38869 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:00:885.040 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:00:885.255 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:00:885.342 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:00:885.408 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:885.420 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692720885399 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:00:885.426 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:00:885.433 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:885.441 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692720885406 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:00:885.446 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:00:885.472 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:885.495 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692720885465 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:00:885.512 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:00:885.521 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:885.535 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692720885509 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:12:00:885.541 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:00:886.388 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:886.411 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:12:00:886.412 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692720886336 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:00:886.807 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:00:886.816 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:886.836 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692720886377 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:00:886.852 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:00:938.575 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:938.632 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692720938545 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:00:938.643 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:00:938.721 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:938.735 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692720938714 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 40929 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:00:938.742 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:00:939.083 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:00:939.174 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:00:939.206 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:00:939.221 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692720939200 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:00:939.228 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:01:179.832 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:01:179.889 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692721179803 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:01:179.900 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:01:179.989 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:01:180.014 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692721179982 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 37679 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:01:180.033 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:01:180.368 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:01:180.533 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:01:180.595 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:01:180.608 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692721180564 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:01:180.615 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:01:420.092 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:01:420.160 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692721419962 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:01:420.173 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:01:420.726 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:01:420.755 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692721420695 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 40303 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:01:420.765 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:01:421.055 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:01:421.115 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:01:421.164 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:01:421.211 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692721421153 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:01:421.239 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:01:947.461 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:01:947.514 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692721947343 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:01:947.526 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:01:947.994 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:01:948.020 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692721947952 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 43769 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:01:948.028 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:01:948.355 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:01:948.423 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:01:948.487 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:01:948.587 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692721948477 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:01:948.594 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:03:221.396 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:03:221.448 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692723221297 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:03:221.461 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:03:221.826 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:03:221.855 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692723221810 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 36851 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:03:221.875 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:03:222.123 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:03:222.196 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:03:222.268 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:03:222.321 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692723222257 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:03:222.328 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:03:829.109 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:03:829.836 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:03:830.678 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692723829003 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:03:830.710 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:03:830.720 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:03:830.733 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692723829515 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 36189 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:03:830.740 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:03:830.747 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:03:830.787 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:03:830.801 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692723830778 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:03:830.816 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:03:830.823 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:03:830.832 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692723830786 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:03:830.838 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:03:830.885 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:03:830.908 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692723830877 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:03:830.919 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:03:831.050 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:03:831.071 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692723831043 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:03:831.079 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:03:831.090 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:03:831.104 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692723831072 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:12:03:831.110 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:03:831.552 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:03:831.568 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692723831545 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:03:831.576 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:03:831.586 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:03:831.603 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692723831569 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:03:831.611 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:03:831.612 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:12:03:882.769 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:03:882.819 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692723882743 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:03:882.829 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:03:882.900 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:03:882.914 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692723882893 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 42545 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:03:882.920 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:03:883.322 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:03:883.418 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:03:883.477 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:03:883.497 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692723883458 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:03:883.505 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:03:982.723 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:03:982.778 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692723982599 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:03:982.789 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:03:982.797 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:03:982.809 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692723982762 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 33551 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:03:982.827 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:03:983.283 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:03:983.374 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:03:983.418 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:03:983.443 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692723983408 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:03:983.451 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:04:909.723 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:04:909.775 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692724909618 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:04:909.788 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:04:910.810 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:04:910.850 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692724910794 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 36421 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:04:910.857 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:04:911.191 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:04:911.304 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:04:911.387 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:04:911.400 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692724911338 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:04:911.419 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:05:392.769 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:05:392.821 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692725392672 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:05:392.833 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:05:393.721 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:05:393.747 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692725393708 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 44927 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:05:393.755 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:05:394.057 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:05:394.179 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:05:394.222 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:05:394.241 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692725394208 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:05:394.248 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:05:394.256 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:05:394.265 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692725394214 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:05:394.355 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:05:394.363 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:05:394.386 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692725394264 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:05:394.408 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:05:394.416 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:05:394.431 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692725394316 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:12:05:394.437 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:05:394.776 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:05:394.791 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692725394745 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:05:394.797 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:05:394.814 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:05:394.829 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692725394808 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:05:394.843 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:05:395.058 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.25 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:12:05:442.351 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:05:442.406 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692725442244 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:05:442.416 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:05:442.707 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:05:442.761 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692725442675 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 41883 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:05:442.775 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:05:443.023 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:05:443.121 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:05:443.176 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:05:443.189 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692725443166 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:05:443.196 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:06:424.405 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:06:424.470 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692726424291 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:06:424.481 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:06:424.889 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:06:424.934 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692726424862 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 33377 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:06:424.943 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:06:425.210 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:06:425.427 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:06:425.464 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:06:425.482 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692726425454 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:06:425.490 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:06:873.560 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:06:873.615 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692726873462 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:06:873.625 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:06:874.342 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:06:874.358 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692726874332 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 40959 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:06:874.373 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:06:874.709 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:06:874.764 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:06:874.856 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:06:874.868 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692726874813 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:06:874.874 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:07:398.496 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:398.548 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692727398386 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:07:398.558 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:07:398.961 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:398.996 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692727398947 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 36383 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:07:399.003 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:07:399.247 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:07:399.311 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:07:399.396 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:399.409 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692727399385 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:07:399.553 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:07:399.561 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:399.582 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692727399395 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:07:399.588 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:07:399.594 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:399.615 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692727399453 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:07:399.626 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:07:399.679 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:399.720 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692727399673 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:07:399.731 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:07:399.746 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:399.760 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692727399688 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:12:07:399.775 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:07:400.201 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:400.217 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692727400171 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:07:400.224 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:07:400.231 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:400.235 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:12:07:400.249 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692727400206 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:07:400.257 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:07:450.415 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:450.474 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692727450291 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:07:450.484 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:07:450.919 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:450.935 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692727450907 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 42427 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:07:450.942 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:07:451.250 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:07:451.329 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:07:451.392 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:451.404 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692727451382 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:07:451.410 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:07:655.730 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:655.800 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692727655552 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:07:655.810 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:07:656.305 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:656.331 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692727656285 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 42353 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:07:656.340 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:07:656.688 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:07:656.791 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:07:656.869 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:657.054 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692727656856 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:07:657.066 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:07:657.075 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:657.085 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692727656866 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:07:657.090 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:07:657.096 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:657.118 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692727657030 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.193, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:07:657.128 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:07:657.136 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:657.150 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692727657089 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:12:07:657.162 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:07:657.780 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:657.810 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692727657722 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:07:657.821 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:07:657.831 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:657.859 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692727657768 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:07:657.872 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:07:657.901 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:12:07:665.613 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:665.675 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692727665495 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 25 dequeues = 25 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 6920 dequeues = 6920 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 26 dequeues = 26 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 26 dequeues = 26 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 6999 dequeues = 6999 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756692698707544 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756692727657830 ] max_sm_queue_count = 7 ] 2025-09-01 Mon 02:12:07:665.697 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:12:07:665.709 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:665.869 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692727665576 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1383 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 122 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1568 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 379 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5328 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1521 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 3471, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6920, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:12:07:665.893 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:12:07:671.439 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:671.491 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692727671415 [SYS_NOTICE]: UveVrouterAgent: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 xmpp_peer_list= [ [ [ ip = 10.20.0.25 status = 0 setup_time = 0 primary = 1 ], [ ip = 10.20.0.193 status = 0 setup_time = 0 primary = 0 ], ] ] dns_servers= [ [ (*_iter149) = 10.20.0.129, (*_iter149) = 10.20.0.25, ] ] ] 2025-09-01 Mon 02:12:07:671.504 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: UveVrouterAgent 2025-09-01 Mon 02:12:07:671.517 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:671.561 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692727671451 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 2242941664 used = 32524 free = 0 buffers = 0 cached = 2303930040 node_type = ] meminfo= [ virt = 4294967295 peakvirt = 18275337 res = 0 ] cpuload= [ one_min_avg = 0.03 five_min_avg = 0.075 fifteen_min_avg = 0.11 ] cpu_share = 1.425 ] ] 2025-09-01 Mon 02:12:07:671.581 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 02:12:07:671.593 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:671.615 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692727671456 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 cpu_info= [ [ [ mem_virt = 4294967295 cpu_share = 1.425 used_sys_mem = 1991069780 one_min_cpuload = 0.03 mem_res = 0 ], ] ] ] 2025-09-01 Mon 02:12:07:671.626 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 02:12:07:671.637 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:671.896 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692727671478 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 in_tpkts = 1953 in_bytes = 238266 out_tpkts = 2044 out_bytes = 114412 exception_packets = 67 exception_packets_allowed = 54 raw_vhost_stats= [ name = vhost0 in_pkts = 1953 in_bytes = 238266 out_pkts = 2044 out_bytes = 114412 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 1 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 1 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 1 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 1 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens6 [ in_pkts = 1861 in_bytes = 106726 out_pkts = 1955 out_bytes = 238318 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens6 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 02:12:07:671.921 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 02:12:07:671.934 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:671.984 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692727671522 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 raw_xmpp_stats= [ [ _iter306->first = 10.20.0.193 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], _iter306->first = 10.20.0.25 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 02:12:07:671.997 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 02:12:07:710.164 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:710.228 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692727710012 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:07:710.239 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:07:710.643 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:710.665 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692727710624 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 40017 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:07:710.674 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:07:710.912 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:07:710.989 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:07:711.059 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:07:711.097 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692727711048 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:07:711.103 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:08:297.669 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:08:297.920 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692728297633 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:08:297.933 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:08:297.945 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:08:297.960 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692728297795 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 39039 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:08:297.979 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:08:298.104 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:08:298.194 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:08:298.254 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:08:298.272 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692728298242 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:08:298.281 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:08:495.576 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:08:495.640 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692728495545 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:08:495.774 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:08:495.786 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:08:495.799 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692728495698 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 42441 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:08:495.805 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:08:495.952 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:08:496.060 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:08:496.117 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:08:496.138 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692728496108 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:08:496.144 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:08:707.655 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 02:12:08:707.961 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Connect : Start Connect timer 10.0.0.249:8086 2025-09-01 Mon 02:12:08:708.066 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:12:08:708.111 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692728707757 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:08:708.132 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:08:708.140 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:12:08:708.228 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 02:12:08:708.261 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: -(-1):EvTcpConnectFail 2025-09-01 Mon 02:12:08:708.279 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692728708053 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1383 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 124 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1571 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 379 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5337 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1524 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 3477, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6936, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:12:08:708.296 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:12:08:708.306 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 02:12:08:708.313 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Connect : EvTcpConnectFail 2025-09-01 Mon 02:12:08:708.626 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:08:708.658 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Idle 2025-09-01 Mon 02:12:08:708.698 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:08:708.819 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692728708345 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1384 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 125 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1571 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 379 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5337 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1524 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 3477, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6938, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:12:08:708.836 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:12:08:708.844 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:08:708.860 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692728708367 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:08:708.867 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:08:708.874 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 02:12:08:708.900 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:08:709.037 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692728708689 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 23614 http_port = 8085 start_time = 1756692067658197 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.131:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1384 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 125 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1572 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionDelete [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 379 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5337 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1524 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 3478, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6938, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:12:08:709.055 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:12:08:710.188 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:08:710.426 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692728710170 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:08:710.440 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:08:710.458 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:08:710.527 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:08:710.564 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:08:710.852 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692728710271 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 33587 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:08:710.872 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:08:710.885 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:08:710.900 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692728710589 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:08:710.908 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:10:314.373 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:10:314.436 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692730314237 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:10:314.446 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:10:314.859 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:10:314.886 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692730314842 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 32823 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:10:314.893 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:10:315.161 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:10:315.226 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:10:315.286 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:10:315.331 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692730315274 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:10:315.338 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:10:618.937 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:10:619.003 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692730618711 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:10:619.014 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:10:619.297 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:10:619.322 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692730619273 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect event = Connect Event local_port = 43075 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:10:619.329 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:10:619.578 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.25 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:10:619.604 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:10:619.650 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692730619559 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:10:619.669 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:10:619.697 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:10:619.750 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:10:619.778 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692730619662 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 38903 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:10:619.784 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:10:619.791 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:10:619.799 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692730619743 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.25 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:10:619.804 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:10:620.017 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:10:620.125 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:10:620.157 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:10:620.298 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692730620148 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:10:620.307 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:10:620.315 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:10:620.325 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692730620153 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:10:620.330 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:10:620.343 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:10:620.363 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692730620211 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = Connect ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:10:620.374 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:10:620.623 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:10:620.655 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692730620614 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:5269, ] ] status = Down description = ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.193, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:10:620.665 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:10:620.674 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:10:620.688 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692730620640 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:12:10:620.701 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:10:621.198 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:10:621.219 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692730621134 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:10:621.226 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:10:621.233 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:10:621.251 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692730621177 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:10:621.260 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:10:621.303 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2:10.20.0.193 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2 To : network-control@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:12:10:668.525 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:10:668.575 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692730668414 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:10:668.585 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:10:668.729 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:10:668.797 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692730668693 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 40075 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:10:668.811 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:10:669.049 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:10:669.103 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:10:669.190 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:10:669.205 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692730669149 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:10:669.211 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:11:722.400 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:11:722.502 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692731722297 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:11:722.514 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:11:723.262 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:11:723.290 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692731723250 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 46443 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:11:723.297 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:11:723.621 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:11:723.814 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:11:723.882 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:11:723.927 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692731723845 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:11:723.937 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:12:222.468 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:12:222.521 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: SANDESH: Send FAILED: 1756692732222364 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:12:222.531 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691734099456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:12:223.266 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:12:223.293 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692732223256 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 35195 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:12:223.300 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:12:223.626 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.129 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:12:223.690 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:12:223.772 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:12:223.784 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692732223727 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:12:223.791 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:12:223.797 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:12:223.805 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692732223731 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:12:223.811 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:12:224.428 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:12:224.472 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692732223849 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:53, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.129, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:12:224.483 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:12:224.516 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:12:224.532 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692732223889 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self Number of connections:4, Expected: 5 ], ] ] ] 2025-09-01 Mon 02:12:12:224.538 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:12:224.546 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:12:224.555 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692732224373 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Idle next_state = Active event = Start ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:12:224.561 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:12:224.567 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:12:224.580 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692732224413 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-ansible-os-3995-2 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.20.0.129 server_addrs= [ [ (*_iter6) = 10.20.0.129:5269, ] ] status = Down description = Idle ], [ type = XMPP name = control-node:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:5269, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.193 server_addrs= [ [ (*_iter6) = 10.20.0.193:53, ] ] status = Down description = Idle ], [ type = XMPP name = dns-server:10.20.0.25 server_addrs= [ [ (*_iter6) = 10.20.0.25:53, ] ] status = Down description = Idle ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.20.0.129, XMPP:control-node:10.20.0.25, XMPP:dns-server:10.20.0.193, XMPP:dns-server:10.20.0.25, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:12:12:224.596 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:12:12:224.580 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: XMPP [SYS_INFO]: XmppConnectionDelete: XMPP Peer an-jenkins-deploy-platform-ansible-os-3995-2/dns:10.20.0.129 Deleted Xmpp Client connection from an-jenkins-deploy-platform-ansible-os-3995-2/dns To : network-dns@contrailsystems.com controller/src/xmpp/xmpp_connection.cc 871 2025-09-01 Mon 02:12:12:274.649 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:12:274.704 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692732274619 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:12:274.714 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:12:274.827 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:12:274.850 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: SANDESH: Send FAILED: 1756692732274817 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 44479 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:12:274.856 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691746694656, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:12:275.234 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:12:275.325 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:12:275.433 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:12:275.454 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692732275378 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:12:275.476 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:13:328.625 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:13:328.681 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: SANDESH: Send FAILED: 1756692733328591 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:13:328.691 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:13:329.157 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:13:329.197 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: SANDESH: Send FAILED: 1756692733329126 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect event = Connect Event local_port = 38289 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:13:329.206 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691729901056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:13:329.482 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691766278784, Pid 23614]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.20.0.193 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:12:13:329.537 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691742496256, Pid 23614]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:12:13:329.585 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:13:329.605 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: SANDESH: Send FAILED: 1756692733329573 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.193 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:13:329.615 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691755091456, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:13:812.106 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:13:812.157 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: SANDESH: Send FAILED: 1756692733811992 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:12:13:812.167 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691750893056, Pid 23614]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:12:13:812.620 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:12:13:812.645 UTC an-jenkins-deploy-platform-ansible-os-3995-2 [Thread 139691759289856, Pid 23614]: SANDESH: Send FAILED: 1756692733812604 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.20.0.129 state = Connect event = Connect Event local_port = 41345 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709