2025-09-01 Mon 02:16:18:664.372 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:16:18:664.437 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:16:18:664.493 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:16:18:664.542 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:18:664.553 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756692978664533 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:16:18:664.610 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:16:18:664.617 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:18:664.631 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756692978664541 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:16:18:664.636 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:16:22:229.562 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:22:229.655 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756692982229505 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:16:22:229.665 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:16:22:230.071 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:22:230.109 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756692982230046 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 45471 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:16:22:230.124 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:16:22:230.307 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:16:22:230.352 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:16:22:230.408 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:22:230.427 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756692982230398 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:16:22:230.433 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:16:22:230.438 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:22:230.448 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756692982230406 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:16:22:230.453 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:16:25:403.885 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:25:403.982 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756692985403686 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 81 dequeues = 81 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 1388 dequeues = 1388 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 81 dequeues = 81 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 82 dequeues = 82 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 1635 dequeues = 1635 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756692957491310 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756692982230438 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 02:16:25:406.367 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:16:25:406.395 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:25:406.545 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756692985403832 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 169 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 320 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 332 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 77 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 115 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 664 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 501, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1388, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:16:25:406.586 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:16:25:410.457 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:25:410.509 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756692985410427 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 0 used = 0 free = 1680651264 buffers = 32730 cached = 1771028152 node_type = ] meminfo= [ virt = 0 peakvirt = 32 res = 0 ] cpuload= [ one_min_avg = 0.14 five_min_avg = 0.075 fifteen_min_avg = 0.055 ] cpu_share = 0.675 ] ] 2025-09-01 Mon 02:16:25:410.520 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 02:16:25:410.534 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:25:410.552 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756692985410450 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 0 cpu_share = 0.675 used_sys_mem = 2523906414 one_min_cpuload = 0.14 mem_res = 0 ], ] ] ] 2025-09-01 Mon 02:16:25:410.561 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 02:16:25:410.577 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:25:410.742 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756692985410481 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 16715 in_bytes = 2977792 out_tpkts = 15292 out_bytes = 2722832 raw_vhost_stats= [ name = vhost0 in_pkts = 16715 in_bytes = 2977792 out_pkts = 15292 out_bytes = 2722832 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 14938 in_bytes = 2707964 out_pkts = 16728 out_bytes = 2978135 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 02:16:25:410.761 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 02:16:25:410.770 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:25:410.800 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756692985410509 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 02:16:25:410.807 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 02:16:27:491.396 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 02:16:27:491.692 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 02:16:27:491.821 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:16:27:491.856 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756692987491497 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:16:27:491.879 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:16:27:491.889 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:16:27:491.973 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 02:16:27:492.001 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 02:16:27:492.007 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756692987491804 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 169 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 322 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 332 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 78 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 117 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 664 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 501, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1394, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:16:27:492.025 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:16:27:492.032 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 02:16:27:492.038 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 02:16:27:492.079 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:16:27:492.105 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Idle 2025-09-01 Mon 02:16:27:492.147 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:27:492.160 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756692987492058 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:16:27:492.169 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:16:27:492.178 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 02:16:27:492.210 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:27:492.301 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756692987492140 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 170 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 323 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 333 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 78 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 117 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 664 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 502, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1396, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:16:27:492.313 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:16:49:174.694 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:49:174.767 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756693009174660 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:16:49:174.786 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:16:49:175.192 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:49:175.231 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693009175163 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 35035 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:16:49:175.239 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:16:49:175.418 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:16:49:175.460 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:16:49:175.500 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:49:175.521 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756693009175489 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:16:49:175.527 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:16:49:175.533 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:49:175.554 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756693009175495 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:16:49:175.559 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:16:52:485.850 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:57176 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 02:16:52:490.562 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:57186 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 02:16:53:010.568 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:53:010.641 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756693013010526 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:16:53:010.651 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:16:53:010.969 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:53:010.983 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756693013010962 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 38633 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:16:53:011.002 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:16:53:011.193 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:16:53:011.271 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:16:53:011.314 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:53:011.335 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756693013011302 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:16:53:011.342 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:16:53:011.349 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:53:011.356 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756693013011317 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:16:53:011.374 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:16:55:404.115 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:55:404.213 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756693015403968 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 82 dequeues = 82 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 1406 dequeues = 1406 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 82 dequeues = 82 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 83 dequeues = 83 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 1656 dequeues = 1656 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756692987492104 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756693013011348 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 02:16:55:404.231 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:16:55:404.241 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:55:404.419 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756693015404081 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 171 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 324 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 337 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 78 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 117 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 672 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 168 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 508, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1406, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:16:55:404.448 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:16:55:410.682 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:55:410.861 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693015410665 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 16867 in_bytes = 3002510 out_tpkts = 15437 out_bytes = 2746174 exception_packets = 76 exception_packets_allowed = 59 raw_vhost_stats= [ name = vhost0 in_pkts = 16867 in_bytes = 3002510 out_pkts = 15437 out_bytes = 2746174 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 3 ] total_in_bandwidth_utilization = 3.1637e-14 total_out_bandwidth_utilization = 2.76472e-14 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 5836, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 5100, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 15080 in_bytes = 2731180 out_pkts = 16880 out_bytes = 3002853 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 0 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 0 ], ] ] ] 2025-09-01 Mon 02:16:55:410.883 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 02:16:55:410.893 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:55:410.921 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693015410697 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 02:16:55:410.933 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 02:16:57:492.194 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 02:16:57:494.165 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 02:16:57:494.256 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:16:57:494.278 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756693017494022 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:16:57:494.285 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:16:57:494.329 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:16:57:494.355 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 02:16:57:494.381 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 02:16:57:494.436 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756693017494241 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 171 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 326 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 337 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 79 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 118 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 672 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 168 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 508, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1410, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:16:57:494.459 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:16:57:494.467 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 02:16:57:494.475 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 02:16:57:494.507 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:16:57:494.532 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Idle 2025-09-01 Mon 02:16:57:494.582 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:57:494.596 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756693017494491 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:16:57:494.601 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:16:57:494.607 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 02:16:57:494.630 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:16:57:494.728 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756693017494563 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 172 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 327 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 338 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 79 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 118 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 672 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 168 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 509, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1412, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:16:57:494.744 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:17:19:145.675 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:19:145.771 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756693039145642 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:17:19:145.782 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:17:19:145.810 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:19:145.825 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756693039145803 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 40097 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:17:19:145.845 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:17:19:146.015 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:17:19:146.059 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:17:19:146.110 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:19:146.158 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756693039146103 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:17:19:146.165 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:17:19:146.170 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:19:146.177 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756693039146107 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:17:19:146.182 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:17:24:061.485 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:24:061.555 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756693044061457 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:17:24:061.565 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:17:24:061.922 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:24:061.948 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693044061915 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 35827 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:17:24:061.954 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:17:24:062.112 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:17:24:062.338 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:17:24:062.372 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:24:062.383 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693044062366 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:17:24:062.387 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:17:24:062.393 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:24:062.400 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693044062370 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:17:24:062.404 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:17:25:404.315 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:25:404.398 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756693045404170 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 83 dequeues = 83 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 1422 dequeues = 1422 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 83 dequeues = 83 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 84 dequeues = 84 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 1675 dequeues = 1675 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756693017494532 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756693044062392 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 02:17:25:404.410 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:17:25:404.418 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:25:404.554 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756693045404284 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 173 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 328 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 340 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 79 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 118 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 680 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 170 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 513, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1422, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:17:25:404.599 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:17:25:410.857 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:25:410.924 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756693045410829 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 32 used = 0 free = 0 buffers = 0 cached = 1771028152 node_type = ] meminfo= [ virt = 32730 peakvirt = 0 res = 0 ] cpuload= [ one_min_avg = 0.265 five_min_avg = 0.13 fifteen_min_avg = 0.07 ] cpu_share = 1.14167 ] ] 2025-09-01 Mon 02:17:25:410.934 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 02:17:25:410.941 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:25:410.954 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756693045410844 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 32730 cpu_share = 1.14167 used_sys_mem = 2523939144 one_min_cpuload = 0.265 mem_res = 0 ], ] ] ] 2025-09-01 Mon 02:17:25:410.960 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 02:17:25:410.966 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:25:411.053 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756693045410876 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 16998 in_bytes = 3017331 out_tpkts = 15556 out_bytes = 2766366 raw_vhost_stats= [ name = vhost0 in_pkts = 16998 in_bytes = 3017331 out_pkts = 15556 out_bytes = 2766366 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 239 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 242 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 239 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 242 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 15196 in_bytes = 2751246 out_pkts = 17011 out_bytes = 3017674 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 239 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 239 ], ] ] ] 2025-09-01 Mon 02:17:25:411.075 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 02:17:25:411.081 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:25:411.100 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756693045410904 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 02:17:25:411.105 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 02:17:27:494.642 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 02:17:27:495.390 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 02:17:27:495.615 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 02:17:27:496.960 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 02:17:27:497.137 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:17:27:497.165 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756693047494789 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:17:27:497.173 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:17:27:497.181 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 02:17:27:497.187 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 02:17:27:497.232 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:17:27:497.271 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Idle 2025-09-01 Mon 02:17:27:497.324 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:27:497.466 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756693047497107 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 173 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 330 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 340 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 80 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 120 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 680 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 170 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 513, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1428, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:17:27:497.487 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:17:27:497.497 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:27:497.511 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756693047497210 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:17:27:497.519 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:17:27:497.525 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 02:17:27:497.557 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:27:497.709 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756693047497314 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 174 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 330 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 341 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 80 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 120 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 680 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 170 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 514, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1429, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:17:27:497.737 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:17:50:466.281 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:50:466.351 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756693070466249 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:17:50:466.362 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:17:50:466.430 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:50:466.444 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756693070466422 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 44823 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:17:50:466.449 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:17:50:466.647 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:17:50:466.703 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:17:50:466.765 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:50:466.775 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756693070466758 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:17:50:466.802 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:17:50:466.808 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:50:466.815 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756693070466765 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:17:50:466.820 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:17:52:507.596 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:42416 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 02:17:52:512.671 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:42420 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 02:17:54:842.546 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:54:842.610 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693074842513 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:17:54:842.619 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:17:54:843.028 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:54:843.070 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756693074842999 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 45421 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:17:54:843.079 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:17:54:843.214 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:17:54:843.260 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:17:54:843.300 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:54:843.321 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693074843292 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:17:54:843.328 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:17:54:843.352 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:54:843.361 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693074843297 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:17:54:843.367 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:17:55:404.500 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:55:404.580 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756693075404384 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 84 dequeues = 84 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 1440 dequeues = 1440 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 84 dequeues = 84 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 85 dequeues = 85 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 1696 dequeues = 1696 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756693047497270 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756693074843351 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 02:17:55:404.594 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:17:55:404.603 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:55:404.715 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756693075404478 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 175 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 332 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 345 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 80 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 120 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 688 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 172 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 520, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1440, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:17:55:404.741 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:17:55:411.069 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:55:411.216 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756693075411037 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 17156 in_bytes = 3042709 out_tpkts = 15715 out_bytes = 2790513 exception_packets = 77 exception_packets_allowed = 60 raw_vhost_stats= [ name = vhost0 in_pkts = 17156 in_bytes = 3042709 out_pkts = 15715 out_bytes = 2790513 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 1181 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 1184 ] phy_if_5min_usage= [ [ [ name = ens3 in_bandwidth_usage = 7826 out_bandwidth_usage = 9361 ], ] ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 1181 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 1184 ] total_in_bandwidth_utilization = 3.18647e-14 total_out_bandwidth_utilization = 2.90837e-14 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 5878, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 5365, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 15352 in_bytes = 2775267 out_pkts = 17170 out_bytes = 3043094 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 1181 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 1181 ], ] ] ] 2025-09-01 Mon 02:17:55:411.241 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 02:17:55:411.252 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:55:411.274 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756693075411071 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 02:17:55:411.284 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 02:17:57:497.313 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 02:17:57:497.545 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 02:17:57:497.716 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 02:17:57:497.742 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 02:17:57:497.742 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:17:57:497.941 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693077497381 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:17:57:497.950 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:17:57:497.958 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:17:57:498.067 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693077497706 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 175 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 334 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 345 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 81 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 121 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 688 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 172 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 520, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1444, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:17:57:498.093 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:17:57:498.100 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 02:17:57:498.106 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 02:17:57:498.144 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:17:57:498.172 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Idle 2025-09-01 Mon 02:17:57:498.222 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:57:498.235 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693077498124 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:17:57:498.241 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:17:57:498.247 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 02:17:57:498.273 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:17:57:498.362 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693077498210 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 176 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 335 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 346 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 81 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 121 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 688 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 172 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 521, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1446, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:17:57:498.375 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:18:21:786.929 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:21:787.258 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:18:21:788.432 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756693101786900 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:18:21:788.441 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:18:21:788.475 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:21:788.486 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756693101787062 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 38801 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:18:21:788.491 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:18:21:788.460 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:18:21:788.545 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:21:788.587 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756693101788538 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:18:21:788.594 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:18:21:788.600 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:21:788.608 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756693101788542 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:18:21:788.612 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:18:22:923.395 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:22:923.451 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756693102923363 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:18:22:923.460 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:18:22:923.543 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:22:923.557 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756693102923536 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 36237 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:18:22:923.592 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:18:22:923.754 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:18:22:923.826 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:18:22:923.861 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:22:923.877 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756693102923853 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:18:22:923.884 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:18:22:923.890 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:22:923.898 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: SANDESH: Send FAILED: 1756693102923858 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:18:22:923.902 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575981254144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:18:25:404.719 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:25:404.795 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756693105404597 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 85 dequeues = 85 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 1456 dequeues = 1456 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 85 dequeues = 85 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 86 dequeues = 86 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 1715 dequeues = 1715 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756693077498171 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756693102923889 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 02:18:25:404.808 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:18:25:404.815 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:25:404.919 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756693105404689 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 177 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 336 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 348 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 81 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 121 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 696 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 174 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 525, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1456, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:18:25:404.951 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:18:25:411.211 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:25:411.278 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756693105411181 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 32 used = 0 free = 0 buffers = 0 cached = 1771028152 node_type = ] meminfo= [ virt = 32730 peakvirt = 0 res = 0 ] cpuload= [ one_min_avg = 0.095 five_min_avg = 0.105 fifteen_min_avg = 0.065 ] cpu_share = 0.658333 ] ] 2025-09-01 Mon 02:18:25:411.289 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 02:18:25:411.296 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:25:411.309 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756693105411192 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 32730 cpu_share = 0.658333 used_sys_mem = 2523939144 one_min_cpuload = 0.095 mem_res = 0 ], ] ] ] 2025-09-01 Mon 02:18:25:411.315 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 02:18:25:411.324 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:25:411.451 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756693105411224 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 17286 in_bytes = 3056173 out_tpkts = 15887 out_bytes = 2812950 exception_packets = 78 exception_packets_allowed = 61 raw_vhost_stats= [ name = vhost0 in_pkts = 17286 in_bytes = 3056173 out_pkts = 15887 out_bytes = 2812950 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 1678 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 1681 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 1678 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 1681 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 15521 in_bytes = 2797578 out_pkts = 17300 out_bytes = 3056558 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 1678 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 1678 ], ] ] ] 2025-09-01 Mon 02:18:25:411.473 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 02:18:25:411.486 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:25:411.507 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756693105411249 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 02:18:25:411.512 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 02:18:27:498.264 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 02:18:27:498.530 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 02:18:27:498.657 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:18:27:498.688 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756693107498366 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:18:27:498.695 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:18:27:498.749 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 02:18:27:498.840 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 02:18:27:498.854 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:18:27:498.963 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756693107498640 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 177 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 338 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 348 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 82 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 123 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 696 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 174 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 525, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1462, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:18:27:498.981 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:18:27:498.995 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 02:18:27:499.000 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 02:18:27:499.037 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:18:27:499.066 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Idle 2025-09-01 Mon 02:18:27:499.108 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:27:499.121 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756693107499018 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:18:27:499.130 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:18:27:499.136 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 02:18:27:499.156 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:27:499.246 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756693107499101 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 178 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 339 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 349 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 82 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 123 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 696 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 174 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 526, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1464, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:18:27:499.259 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:18:50:138.703 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:50:138.782 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756693130138664 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:18:50:138.795 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:18:50:138.843 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:50:138.868 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756693130138826 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 33427 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:18:50:138.875 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:18:50:138.990 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:18:50:139.051 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:18:50:139.116 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:50:139.137 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756693130139094 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:18:50:139.144 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:18:50:139.150 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:50:139.157 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756693130139107 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:18:50:139.161 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:18:51:274.132 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:51:274.225 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: SANDESH: Send FAILED: 1756693131274077 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:18:51:274.239 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575960262144, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:18:51:274.751 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:51:274.797 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: SANDESH: Send FAILED: 1756693131274705 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 36147 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:18:51:274.807 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575989650944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:18:51:274.933 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:18:51:274.988 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:18:51:275.040 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:51:275.068 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756693131275026 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:18:51:275.076 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:18:51:275.105 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:51:275.115 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756693131275035 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:18:51:275.121 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:18:52:534.604 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:47558 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 02:18:52:539.050 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.50:47562 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-09-01 Mon 02:18:55:405.049 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:55:405.139 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756693135404867 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 86 dequeues = 86 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 1474 dequeues = 1474 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 86 dequeues = 86 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 87 dequeues = 87 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 1736 dequeues = 1736 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756693107499065 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756693131275104 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 02:18:55:405.154 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:18:55:405.161 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:55:405.304 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756693135405013 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 179 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 340 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 353 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 82 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 123 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 704 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 176 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 532, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1474, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:18:55:405.337 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:18:55:411.469 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:55:411.651 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693135411441 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 17869 in_bytes = 3203564 out_tpkts = 16478 out_bytes = 2895762 raw_vhost_stats= [ name = vhost0 in_pkts = 17869 in_bytes = 3203564 out_pkts = 16478 out_bytes = 2895762 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 2214 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 2217 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 2214 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 2217 ] total_in_bandwidth_utilization = 7.58887e-14 total_out_bandwidth_utilization = 1.16264e-13 flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] phy_band_in_bps= [ [ _iter260->first = ens3 _iter260->second = 13999, ] ] phy_band_out_bps= [ [ _iter261->first = ens3 _iter261->second = 21447, ] ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 16109 in_bytes = 2880264 out_pkts = 17883 out_bytes = 3203949 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 2214 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 2214 ], ] ] ] 2025-09-01 Mon 02:18:55:411.681 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 02:18:55:411.693 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:55:411.715 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693135411484 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 02:18:55:411.721 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 02:18:57:499.185 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 02:18:57:499.526 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 02:18:57:499.683 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:18:57:499.712 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756693137499298 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:18:57:499.720 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:18:57:499.727 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:18:57:499.751 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 02:18:57:499.779 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 02:18:57:499.841 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756693137499666 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 179 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 342 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 353 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 83 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 124 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 704 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 176 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 532, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1478, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:18:57:499.880 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:18:57:499.905 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 02:18:57:499.912 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 02:18:57:499.949 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:18:57:499.980 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Idle 2025-09-01 Mon 02:18:57:500.020 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:57:500.034 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756693137499931 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:18:57:500.040 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:18:57:500.045 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 02:18:57:500.075 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:18:57:500.167 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: SANDESH: Send FAILED: 1756693137500011 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 180 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 343 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 354 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 83 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 124 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 704 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 176 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 533, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1480, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:18:57:500.181 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575968658944, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:19:18:489.236 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:19:18:489.377 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756693158489204 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:19:18:489.392 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:19:18:489.409 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:19:18:489.424 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756693158489352 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 39647 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:19:18:489.430 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:19:18:489.649 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-dns@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:19:18:489.723 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:19:18:489.783 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:19:18:489.818 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693158489756 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:19:18:489.825 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:19:18:489.831 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:19:18:489.839 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693158489775 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:19:18:489.844 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:19:18:815.191 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:19:18:815.249 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693158815158 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active next_state = Connect event = EvConnectTimerExpired ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:19:18:815.259 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:19:18:815.587 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:19:18:815.618 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: SANDESH: Send FAILED: 1756693158815578 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect event = Connect Event local_port = 36307 remote_port = 4294967295 ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:19:18:815.624 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:19:18:815.741 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer RECV Mode Client: Event: Tcp Connect Fail peer ip: 10.0.0.50 ( network-control@contrailsystems.com ) controller/src/xmpp/xmpp_state_machine.cc 1299 2025-09-01 Mon 02:19:18:815.837 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575985452544, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session - > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:19:18:815.871 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:19:18:815.889 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756693158815862 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Connect next_state = Active event = EvTcpConnectFail ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:19:18:815.894 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:19:18:815.900 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:19:18:815.907 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756693158815867 [SYS_INFO]: XmppConnectionLog: connection= [ ip_address = 10.0.0.50 state = Active event = Connect failed after retries ] file = controller/src/xmpp/xmpp_state_machine.cc line = 1709 2025-09-01 Mon 02:19:18:815.912 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: XmppConnectionLog 2025-09-01 Mon 02:19:25:405.251 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:19:25:405.343 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693165405098 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 87 dequeues = 87 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 1490 dequeues = 1490 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 87 dequeues = 87 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 88 dequeues = 88 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 1755 dequeues = 1755 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1756693137499979 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1756693158815899 ] max_sm_queue_count = 4 ] 2025-09-01 Mon 02:19:25:405.357 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:19:25:405.379 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:19:25:405.503 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693165405192 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 181 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 344 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 356 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 83 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 124 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 712 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 178 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 537, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1490, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:19:25:405.521 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:19:25:411.627 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:19:25:411.683 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756693165411602 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ num_cpu = 2 sys_mem_info= [ total = 32 used = 0 free = 0 buffers = 0 cached = 1771028152 node_type = ] meminfo= [ virt = 32730 peakvirt = 0 res = 0 ] cpuload= [ one_min_avg = 0.085 five_min_avg = 0.105 fifteen_min_avg = 0.07 ] cpu_share = 0.716667 ] ] 2025-09-01 Mon 02:19:25:411.691 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 02:19:25:411.699 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:19:25:411.713 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756693165411615 [SYS_NOTICE]: ComputeCpuStateTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 cpu_info= [ [ [ mem_virt = 32730 cpu_share = 0.716667 used_sys_mem = 2523939144 one_min_cpuload = 0.085 mem_res = 0 ], ] ] ] 2025-09-01 Mon 02:19:25:411.719 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: ComputeCpuStateTrace 2025-09-01 Mon 02:19:25:411.726 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:19:25:411.811 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756693165411642 [SYS_NOTICE]: VrouterStats: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 in_tpkts = 18064 in_bytes = 3230094 out_tpkts = 16763 out_bytes = 2935405 raw_vhost_stats= [ name = vhost0 in_pkts = 18064 in_bytes = 3230094 out_pkts = 16763 out_bytes = 2935405 speed = -1 duplexity = -1 ] raw_vhost_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 2640 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 2643 ] raw_drop_stats= [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 2640 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 3 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 2643 ] flow_rate= [ added_flows = 0 max_flow_adds_per_second = 0 min_flow_adds_per_second = 0 deleted_flows = 0 max_flow_deletes_per_second = 0 min_flow_deletes_per_second = 0 active_flows = 0 hold_flows = 0 ] raw_phy_if_stats= [ [ _iter262->first = ens3 [ in_pkts = 16391 in_bytes = 2919781 out_pkts = 18078 out_bytes = 3230479 ], ] ] raw_phy_if_drop_stats= [ [ _iter263->first = ens3 [ ds_discard = 0 ds_pull = 0 ds_invalid_if = 0 ds_invalid_arp = 0 ds_trap_no_if = 0 ds_nowhere_to_go = 0 ds_flow_queue_limit_exceeded = 0 ds_flow_no_memory = 0 ds_flow_invalid_protocol = 0 ds_flow_nat_no_rflow = 0 ds_flow_action_drop = 0 ds_flow_action_invalid = 0 ds_flow_unusable = 0 ds_flow_table_full = 0 ds_interface_tx_discard = 0 ds_interface_drop = 2640 ds_duplicated = 0 ds_push = 0 ds_ttl_exceeded = 0 ds_invalid_nh = 0 ds_invalid_label = 0 ds_invalid_protocol = 0 ds_interface_rx_discard = 0 ds_invalid_mcast_source = 0 ds_head_alloc_fail = 0 ds_pcow_fail = 0 ds_mcast_clone_fail = 0 ds_rewrite_fail = 0 ds_misc = 0 ds_invalid_packet = 0 ds_cksum_err = 0 ds_no_fmd = 0 ds_invalid_vnid = 0 ds_frag_err = 0 ds_invalid_source = 0 ds_mcast_df_bit = 0 ds_l2_no_route = 0 ds_vlan_fwd_tx = 0 ds_vlan_fwd_enq = 0 ds_drop_new_flow = 0 ds_flow_evict = 0 ds_trap_original = 0 ds_no_memory = 0 ds_fragment_queue_fail = 0 ds_no_frag_entry = 0 ds_icmp_error = 0 ds_clone_fail = 0 ds_drop_pkts = 2640 ], ] ] ] 2025-09-01 Mon 02:19:25:411.829 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterStats 2025-09-01 Mon 02:19:25:411.835 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:19:25:411.855 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: SANDESH: Send FAILED: 1756693165411664 [SYS_NOTICE]: VrouterControlStatsTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 raw_xmpp_stats= [ [ _iter306->first = 10.0.0.50 [ reconnects = 0 in_msgs = 0 out_msgs = 0 ], ] ] raw_ifmap_stats= [ [ _iter307->first = link_delete_parse_errors _iter307->second = 0, _iter307->first = link_update_parse_errors _iter307->second = 0, _iter307->first = node_delete_parse_errors _iter307->second = 0, _iter307->first = node_update_parse_errors _iter307->second = 0, ] ] raw_rt_table_size= [ [ _iter308->first = default-domain:default-project:ip-fabric:__default__ [ inet4_unicast = 7 inet4_multicast = 0 evpn = 0 bridge = 5 inet6_unicast = 2 ], _iter308->first = default-domain:default-project:ip-fabric:ip-fabric [ inet4_unicast = 2 inet4_multicast = 0 evpn = 1 bridge = 2 inet6_unicast = 0 ], ] ] ] 2025-09-01 Mon 02:19:25:411.864 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575964460544, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: VrouterControlStatsTrace 2025-09-01 Mon 02:19:27:500.024 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-09-01 Mon 02:19:27:500.356 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Connect : Start Connect timer 10.0.0.50:8086 2025-09-01 Mon 02:19:27:500.453 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:19:27:500.519 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693167500095 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:19:27:500.528 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:19:27:500.536 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Connect 2025-09-01 Mon 02:19:27:500.538 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1) OnSessionEvent TCP Connect Failed 2025-09-01 Mon 02:19:27:500.583 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: -(-1):EvTcpConnectFail 2025-09-01 Mon 02:19:27:500.656 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693167500441 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Connect successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 181 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 346 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 356 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 84 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 126 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 712 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 178 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 537, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1496, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:19:27:500.683 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:19:27:500.695 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvTcpConnectFail in state Connect 2025-09-01 Mon 02:19:27:500.700 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Connect : EvTcpConnectFail 2025-09-01 Mon 02:19:27:500.735 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-09-01 Mon 02:19:27:500.763 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Idle 2025-09-01 Mon 02:19:27:500.803 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:19:27:500.820 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693167500718 [SYS_NOTICE]: NodeStatusUVE: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1 process_status= [ [ [ module_id = contrail-vrouter-agent instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = XMPP name = control-node:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:5269, ] ] status = Down description = Connect ], [ type = XMPP name = dns-server:10.0.0.50 server_addrs= [ [ (*_iter6) = 10.0.0.50:53, ] ] status = Down description = Connect ], [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], ] ] flag_infos= [ [ ] ] description = XMPP:control-node:10.0.0.50, XMPP:dns-server:10.0.0.50, Collector connection down, No Configuration for self ], ] ] ] 2025-09-01 Mon 02:19:27:500.825 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-09-01 Mon 02:19:27:500.831 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvTcpDeleteSession in state Idle 2025-09-01 Mon 02:19:27:500.853 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Processing scm::EvSandeshSend in state Idle 2025-09-01 Mon 02:19:27:500.945 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: SANDESH: Send FAILED: 1756693167500796 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = an-jenkins-deploy-platform-juju-k8s-1454-1:Compute:contrail-vrouter-agent:0 client_info= [ status = Idle successful_connections = 0 pid = 147334 http_port = 8085 start_time = 1756690645384527 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = ComputeCpuStateTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 182 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 347 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 357 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVirtualNetworkAgentTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = UveVrouterAgent [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterControlStatsTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 84 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = VrouterStats [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 126 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppConnectionLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 712 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = XmppEventLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 178 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 538, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 1498, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-09-01 Mon 02:19:27:500.957 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575977055744, Pid 147334]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-09-01 Mon 02:19:28:755.705 UTC an-jenkins-deploy-platform-juju-k8s-1454-1 [Thread 140575996639872, Pid 147334]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8085::10.0.0.22:60974 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255