2026-06-08 Mon 05:45:10:543.650 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: http-server-port 8091 2026-06-08 Mon 05:45:10:543.717 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Max-tasks 16 2026-06-08 Mon 05:45:10:543.724 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Max-slice 100 2026-06-08 Mon 05:45:10:543.729 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Collectors 10.0.0.38:8086 2026-06-08 Mon 05:45:10:543.733 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Collectors 10.0.0.50:8086 2026-06-08 Mon 05:45:10:543.737 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Collectors 10.0.0.33:8086 2026-06-08 Mon 05:45:10:543.759 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: SANDESH: ROLE : Generator 2026-06-08 Mon 05:45:10:543.764 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-06-08 Mon 05:45:10:543.768 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5902-2. 2026-06-08 Mon 05:45:10:543.772 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: SANDESH: NODE TYPE : Database 2026-06-08 Mon 05:45:10:543.777 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: SANDESH: INSTANCE ID : 0 2026-06-08 Mon 05:45:10:543.781 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-06-08 Mon 05:45:10:543.786 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-06-08 Mon 05:45:10:546.643 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: HTTP Introspect Init 2026-06-08 Mon 05:45:10:546.745 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2026-06-08 Mon 05:45:10:546.823 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Sandesh Http Server Port: 8091 2026-06-08 Mon 05:45:10:546.877 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-06-08 Mon 05:45:10:546.884 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-06-08 Mon 05:45:10:547.211 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Idle 2026-06-08 Mon 05:45:10:547.348 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: SANDESH: No Client: 1780897510547280 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780897510547222 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-08 Mon 05:45:10:547.479 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-06-08 Mon 05:45:10:547.491 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-06-08 Mon 05:45:10:547.501 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Cassandra Servers: 10.0.0.38:9042 10.0.0.50:9042 10.0.0.33:9042 2026-06-08 Mon 05:45:10:548.949 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636346930752, Pid 1]: Processing scm::EvStart in state Idle 2026-06-08 Mon 05:45:10:549.008 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636346930752, Pid 1]: Disconnect 2026-06-08 Mon 05:45:10:549.167 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636346930752, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-06-08 Mon 05:45:10:550.161 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636346930752, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-06-08 Mon 05:45:10:550.246 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636346930752, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:45:10:550.264 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636346930752, Pid 1]: SANDESH: Send FAILED: 1780897510549003 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-06-08 Mon 05:45:10:550.288 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636346930752, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:45:10:550.308 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636346930752, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:45:10:550.390 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636346930752, Pid 1]: SANDESH: Send FAILED: 1780897510549155 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780897510547222 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-08 Mon 05:45:10:550.406 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636346930752, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:45:10:550.414 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636346930752, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:45:10:550.476 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636346930752, Pid 1]: SANDESH: Send FAILED: 1780897510549974 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780897510547222 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-08 Mon 05:45:10:550.487 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636346930752, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:45:10:550.495 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636346930752, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:45:10:550.522 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636346930752, Pid 1]: SANDESH: Send FAILED: 1780897510550006 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-06-08 Mon 05:45:10:550.528 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636346930752, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:45:10:550.534 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636346930752, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:45:10:550.580 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636346930752, Pid 1]: SANDESH: Send FAILED: 1780897510550234 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780897510547222 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-08 Mon 05:45:10:550.614 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636346930752, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:45:10:552.356 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-06-08 Mon 05:45:10:552.373 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-06-08 Mon 05:45:10:552.429 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-06-08 Mon 05:45:10:553.165 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:45:10:553.282 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: SANDESH: Send FAILED: 1780897510552415 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-06-08 Mon 05:45:10:553.319 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:45:15:554.587 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-06-08 Mon 05:45:15:554.625 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-06-08 Mon 05:45:15:554.650 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-06-08 Mon 05:45:20:557.250 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-06-08 Mon 05:45:20:557.312 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-06-08 Mon 05:45:20:557.337 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-06-08 Mon 05:45:25:559.871 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-06-08 Mon 05:45:25:559.922 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-06-08 Mon 05:45:25:559.946 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-06-08 Mon 05:45:30:606.930 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2026-06-08 Mon 05:45:30:606.967 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-06-08 Mon 05:45:30:606.985 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-06-08 Mon 05:45:35:646.101 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2026-06-08 Mon 05:45:35:646.172 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-06-08 Mon 05:45:35:646.194 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-06-08 Mon 05:45:40:682.612 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2026-06-08 Mon 05:45:40:682.659 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-06-08 Mon 05:45:40:682.677 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-06-08 Mon 05:45:45:748.951 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2026-06-08 Mon 05:45:45:749.026 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-06-08 Mon 05:45:45:749.057 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 7 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-06-08 Mon 05:45:50:784.342 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-06-08 Mon 05:45:50:784.392 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 8 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-06-08 Mon 05:45:55:812.009 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-06-08 Mon 05:46:00:817.621 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:46:00:817.664 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: RAC_Connect status: 0 0x2624820 2026-06-08 Mon 05:46:00:817.691 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:46:00:817.705 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:46:00:817.716 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: RAC_Connect status: 0 0x2625340 2026-06-08 Mon 05:46:00:817.720 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:46:00:817.728 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:46:00:817.735 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: RAC_Connect status: 0 0x2625f00 2026-06-08 Mon 05:46:00:817.739 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:46:00:817.746 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:46:00:817.754 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: RAC_Connect status: 0 0x26269e0 2026-06-08 Mon 05:46:00:817.758 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:46:00:817.765 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:46:00:817.771 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: RAC_Connect status: 0 0x26274c0 2026-06-08 Mon 05:46:00:817.776 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:46:00:817.783 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:46:00:817.798 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: RAC_Connect status: 0 0x261d1a0 2026-06-08 Mon 05:46:00:817.803 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:46:00:817.810 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:46:00:817.817 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: RAC_Connect status: 0 0x2622180 2026-06-08 Mon 05:46:00:817.821 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:46:00:817.828 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:46:00:817.835 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: RAC_Connect status: 0 0x26228e0 2026-06-08 Mon 05:46:00:817.839 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:46:00:817.846 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:46:00:817.853 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: RAC_Connect status: 0 0x26231e0 2026-06-08 Mon 05:46:00:817.857 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:46:00:817.864 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:46:00:817.871 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: RAC_Connect status: 0 0x2623d00 2026-06-08 Mon 05:46:00:817.875 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:46:00:817.882 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:46:00:817.889 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: RAC_Connect status: 0 0x2627fa0 2026-06-08 Mon 05:46:00:817.893 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:46:00:817.900 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:46:00:817.906 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: RAC_Connect status: 0 0x2629560 2026-06-08 Mon 05:46:00:817.911 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:46:00:817.918 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:46:00:817.924 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: RAC_Connect status: 0 0x2628a80 2026-06-08 Mon 05:46:00:817.928 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:46:00:817.935 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:46:00:817.944 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: RAC_Connect status: 0 0x262a040 2026-06-08 Mon 05:46:00:817.949 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:46:00:817.955 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:46:00:817.962 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: RAC_Connect status: 0 0x262ab20 2026-06-08 Mon 05:46:00:817.967 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:46:00:817.973 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:46:00:817.993 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-06-08 Mon 05:46:00:817.999 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: -(-1):EvTcpConnectFail 2026-06-08 Mon 05:46:00:817.680 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: SANDESH: Send FAILED: 1780897560817556 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-06-08 Mon 05:46:00:818.099 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:46:00:818.109 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-06-08 Mon 05:46:00:818.116 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Connect : EvTcpConnectFail 2026-06-08 Mon 05:46:00:818.189 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:46:00:818.202 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:46:00:818.245 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2026-06-08 Mon 05:46:00:818.254 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:46:00:818.269 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:46:00:818.277 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:46:00:818.285 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Idle 2026-06-08 Mon 05:46:00:818.327 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:46:00:818.340 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:46:00:818.348 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:46:00:818.359 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:46:00:818.366 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:46:00:818.377 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:46:00:818.384 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:46:00:818.399 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:46:00:818.459 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:46:00:818.473 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:46:00:818.489 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:46:00:818.496 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:46:00:818.509 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:46:00:818.512 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:46:00:818.520 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:46:00:818.542 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:46:00:818.551 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:46:00:818.562 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:46:00:818.572 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:46:00:818.578 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:46:00:818.588 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:46:00:818.594 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:46:00:818.621 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:46:00:818.628 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:46:00:818.656 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:46:00:818.657 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: SANDESH: Send FAILED: 1780897560818191 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780897510547222 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 42 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 43, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-08 Mon 05:46:00:818.674 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:46:00:818.728 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:46:00:818.744 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:46:00:818.770 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:46:00:818.771 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:46:00:818.800 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:46:00:818.801 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: SANDESH: Send FAILED: 1780897560818219 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-06-08 Mon 05:46:00:818.815 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:46:00:818.842 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:46:00:818.831 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:46:00:818.856 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:46:00:818.882 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:46:00:818.874 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:46:00:818.896 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:46:00:818.920 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: SANDESH: Send FAILED: 1780897560818245 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:46:00:818.929 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:46:00:818.943 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-08 Mon 05:46:00:818.931 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:46:00:818.991 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:46:00:818.991 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:46:00:819.013 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:46:00:819.017 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: SANDESH: Send FAILED: 1780897560818300 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-08 Mon 05:46:00:819.032 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:46:00:819.035 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:46:00:819.047 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:46:00:819.059 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:46:00:819.081 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:46:00:819.090 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:46:00:819.194 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: SANDESH: Send FAILED: 1780897560818476 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780897510547222 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 55 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 57, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-08 Mon 05:46:00:819.216 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:46:00:819.228 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:46:00:819.248 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: SANDESH: Send FAILED: 1780897560818650 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-08 Mon 05:46:00:819.256 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:46:00:819.265 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:46:00:819.286 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: SANDESH: Send FAILED: 1780897560819030 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-08 Mon 05:46:00:819.294 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:46:00:819.434 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:46:00:819.481 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: SANDESH: Send FAILED: 1780897560819360 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1780897560818284 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1780897560819263 ] max_sm_queue_count = 6 ] 2026-06-08 Mon 05:46:00:819.490 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:46:00:819.499 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:46:00:819.611 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: SANDESH: Send FAILED: 1780897560819415 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780897510547222 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 87 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 14, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-08 Mon 05:46:00:819.628 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:46:01:818.480 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635482879552, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-06-08 Mon 05:46:01:818.761 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635482879552, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-06-08 Mon 05:46:01:818.881 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635482879552, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:46:01:818.916 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635482879552, Pid 1]: SANDESH: Send FAILED: 1780897561818581 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-08 Mon 05:46:01:818.935 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635482879552, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:46:01:818.946 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635482879552, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:46:01:819.081 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635482879552, Pid 1]: SANDESH: Send FAILED: 1780897561818862 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780897510547222 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 87 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-08 Mon 05:46:01:819.101 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635482879552, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:46:01:820.658 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:49270::10.0.0.33:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 281 2026-06-08 Mon 05:46:01:820.685 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: 10.0.0.50:49270::10.0.0.33:8086(-1) OnSessionEvent TCP Connected 2026-06-08 Mon 05:46:01:820.694 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: 10.0.0.50:49270::10.0.0.33:8086(-1):EvTcpConnected 2026-06-08 Mon 05:46:01:820.764 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635470284352, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-06-08 Mon 05:46:01:820.780 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635470284352, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-06-08 Mon 05:46:01:820.820 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635470284352, Pid 1]: ClientInit 2026-06-08 Mon 05:46:01:820.840 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635470284352, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5902-2.:contrail-query-engine:0:Database count 1 2026-06-08 Mon 05:46:01:820.935 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635470284352, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:46:01:820.946 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635470284352, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-08 Mon 05:46:01:820.971 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635470284352, Pid 1]: SANDESH: Send FAILED: 1780897561820813 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-08 Mon 05:46:01:820.988 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635470284352, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-08 Mon 05:46:01:820.998 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635470284352, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:46:01:821.004 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635470284352, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-06-08 Mon 05:46:01:821.053 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635470284352, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:46:01:821.059 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635470284352, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-08 Mon 05:46:01:821.133 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635470284352, Pid 1]: SANDESH: Send FAILED: 1780897561820918 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780897510547222 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 87 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-08 Mon 05:46:01:821.149 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635470284352, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-08 Mon 05:46:01:829.601 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636342732352, Pid 1]: OnMessage control in state: ClientInit 2026-06-08 Mon 05:46:01:829.706 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636330137152, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-06-08 Mon 05:46:01:829.737 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636330137152, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-06-08 Mon 05:46:01:829.863 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636330137152, Pid 1]: Received Ctrl Message with size 0 2026-06-08 Mon 05:46:01:830.027 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636330137152, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780897510547222 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1780897560818284 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1780897560819263 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 87 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 6 proxy seq 10 2026-06-08 Mon 05:46:01:830.060 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636330137152, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-06-08 Mon 05:46:01:830.082 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636330137152, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2026-06-08 Mon 05:46:01:830.093 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636330137152, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-06-08 Mon 05:46:01:830.100 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636330137152, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-06-08 Mon 05:46:01:830.105 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636330137152, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-06-08 Mon 05:46:01:830.138 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636330137152, Pid 1]: Established 2026-06-08 Mon 05:46:09:097.545 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: OnMessage control in state: Established 2026-06-08 Mon 05:46:09:097.615 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2026-06-08 Mon 05:46:09:097.623 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Established : EvSandeshMessageRecv 2026-06-08 Mon 05:46:09:097.649 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: Received Ctrl Message with size 3 2026-06-08 Mon 05:46:09:097.663 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 11, total = 0 2026-06-08 Mon 05:46:09:097.672 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 13, total = 0 2026-06-08 Mon 05:46:09:097.683 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2026-06-08 Mon 05:46:09:097.690 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-06-08 Mon 05:46:30:821.702 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636313343552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-06-08 Mon 05:47:30:823.864 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636338533952, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-06-08 Mon 05:48:30:825.574 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636304946752, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-06-08 Mon 05:49:27:029.190 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.50:41552 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2026-06-08 Mon 05:49:30:826.469 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140635478681152, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-06-08 Mon 05:49:38:240.241 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636358271488, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.50:41814 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2026-06-08 Mon 05:50:30:827.577 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636342732352, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-06-08 Mon 05:51:25:589.111 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636317541952, Pid 1]: Processing scm::EvStop in state Established 2026-06-08 Mon 05:51:25:589.203 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636317541952, Pid 1]: Idle 2026-06-08 Mon 05:51:25:589.290 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636317541952, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-08 Mon 05:51:25:589.300 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636317541952, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:51:25:589.395 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636317541952, Pid 1]: SANDESH: Send FAILED: 1780897885589276 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780897510547222 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 283397 calls = 31 average_bytes = 9141 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 87 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 22 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 32, ] ] ] 2026-06-08 Mon 05:51:25:589.429 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140636317541952, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:51:38:557.548 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: http-server-port 8091 2026-06-08 Mon 05:51:38:557.609 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Max-tasks 16 2026-06-08 Mon 05:51:38:557.615 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Max-slice 100 2026-06-08 Mon 05:51:38:557.620 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Collectors 10.0.0.38:8086 2026-06-08 Mon 05:51:38:557.624 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Collectors 10.0.0.50:8086 2026-06-08 Mon 05:51:38:557.627 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Collectors 10.0.0.33:8086 2026-06-08 Mon 05:51:38:557.649 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: SANDESH: ROLE : Generator 2026-06-08 Mon 05:51:38:557.654 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-06-08 Mon 05:51:38:557.658 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5902-2. 2026-06-08 Mon 05:51:38:557.662 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: SANDESH: NODE TYPE : Database 2026-06-08 Mon 05:51:38:557.666 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: SANDESH: INSTANCE ID : 0 2026-06-08 Mon 05:51:38:557.671 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-06-08 Mon 05:51:38:557.676 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-06-08 Mon 05:51:38:560.298 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: HTTP Introspect Init 2026-06-08 Mon 05:51:38:560.393 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2026-06-08 Mon 05:51:38:560.480 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Sandesh Http Server Port: 8091 2026-06-08 Mon 05:51:38:560.518 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-06-08 Mon 05:51:38:560.526 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-06-08 Mon 05:51:38:560.878 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Idle 2026-06-08 Mon 05:51:38:561.009 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: SANDESH: No Client: 1780897898560954 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-08 Mon 05:51:38:563.189 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-06-08 Mon 05:51:38:563.204 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-06-08 Mon 05:51:38:563.217 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Cassandra Servers: 10.0.0.38:9042 10.0.0.50:9042 10.0.0.33:9042 2026-06-08 Mon 05:51:38:563.273 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Processing scm::EvStart in state Idle 2026-06-08 Mon 05:51:38:563.342 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Disconnect 2026-06-08 Mon 05:51:38:563.426 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-06-08 Mon 05:51:38:563.637 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-06-08 Mon 05:51:38:563.713 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:51:38:563.735 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: SANDESH: Send FAILED: 1780897898563336 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-06-08 Mon 05:51:38:563.745 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:51:38:563.756 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:51:38:563.851 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: SANDESH: Send FAILED: 1780897898563398 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-08 Mon 05:51:38:563.871 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:51:38:563.882 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:51:38:563.957 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: SANDESH: Send FAILED: 1780897898563460 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-08 Mon 05:51:38:563.973 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:51:38:563.983 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:51:38:564.000 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: SANDESH: Send FAILED: 1780897898563480 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-06-08 Mon 05:51:38:564.006 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:51:38:564.012 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:51:38:564.051 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: SANDESH: Send FAILED: 1780897898563697 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-08 Mon 05:51:38:564.061 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:51:38:589.639 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-06-08 Mon 05:52:18:613.129 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797404309056, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:52:18:613.190 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797404309056, Pid 1]: SANDESH: Send FAILED: 1780897938613000 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-06-08 Mon 05:52:18:613.199 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797404309056, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:52:18:614.997 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:57232::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 281 2026-06-08 Mon 05:52:18:615.023 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:57232::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2026-06-08 Mon 05:52:18:615.031 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:57232::10.0.0.38:8086(-1):EvTcpConnected 2026-06-08 Mon 05:52:18:615.093 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0f770 2026-06-08 Mon 05:52:18:615.128 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10290 2026-06-08 Mon 05:52:18:615.149 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10e50 2026-06-08 Mon 05:52:18:615.164 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a11930 2026-06-08 Mon 05:52:18:615.168 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-06-08 Mon 05:52:18:615.201 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-06-08 Mon 05:52:18:615.293 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: ClientInit 2026-06-08 Mon 05:52:18:615.315 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5902-2.:contrail-query-engine:0:Database count 1 2026-06-08 Mon 05:52:18:615.483 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:52:18:615.495 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-08 Mon 05:52:18:615.517 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: SANDESH: Send FAILED: 1780897938615254 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-06-08 Mon 05:52:18:615.526 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-08 Mon 05:52:18:615.536 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:52:18:615.550 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-06-08 Mon 05:52:18:615.575 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:52:18:615.582 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-08 Mon 05:52:18:615.727 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: SANDESH: Send FAILED: 1780897938615458 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 3, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-08 Mon 05:52:18:615.748 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-08 Mon 05:52:18:615.175 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a08180 2026-06-08 Mon 05:52:18:616.222 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:52:18:616.260 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:52:18:616.287 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12410 2026-06-08 Mon 05:52:18:616.313 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a0d180 2026-06-08 Mon 05:52:18:616.348 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:52:18:616.363 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:52:18:616.381 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a0d830 2026-06-08 Mon 05:52:18:616.388 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:52:18:616.395 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:52:18:616.411 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a0e130 2026-06-08 Mon 05:52:18:619.245 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:52:18:619.272 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:52:18:619.287 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a0ec50 2026-06-08 Mon 05:52:18:619.292 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:52:18:619.298 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:52:18:619.309 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12ef0 2026-06-08 Mon 05:52:18:619.343 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a139d0 2026-06-08 Mon 05:52:18:619.356 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a144b0 2026-06-08 Mon 05:52:18:619.368 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a14f90 2026-06-08 Mon 05:52:18:619.380 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a15a70 2026-06-08 Mon 05:52:18:619.623 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:52:18:619.771 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:52:18:620.468 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-08 Mon 05:52:18:620.476 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:52:18:620.528 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:52:18:620.536 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:52:18:620.585 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:52:18:620.587 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: SANDESH: Send FAILED: 1780897938619538 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = ClientInit state_since = 1780897938615262 last_state = ClientInit last_event = scm::EvSandeshSend last_event_at = 1780897938615574 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 1 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 5 ] 2026-06-08 Mon 05:52:18:620.621 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-08 Mon 05:52:18:620.625 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:52:18:620.635 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:52:18:620.651 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:52:18:620.658 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-08 Mon 05:52:18:620.672 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:52:18:620.694 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:52:18:620.702 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:52:18:620.712 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:52:18:620.723 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:52:18:620.737 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:52:18:620.750 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:52:18:620.762 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:52:18:620.790 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: SANDESH: Send FAILED: 1780897938619596 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 8, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 8, _iter59->first = sent _iter59->second = 1, ] ] ] 2026-06-08 Mon 05:52:18:620.812 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-08 Mon 05:52:18:620.821 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:52:18:620.827 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-08 Mon 05:52:18:620.842 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: SANDESH: Send FAILED: 1780897938620575 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:52:18:620.850 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-08 Mon 05:52:18:620.860 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:52:18:620.867 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-08 Mon 05:52:18:620.890 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: SANDESH: Send FAILED: 1780897938620611 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-08 Mon 05:52:18:620.898 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-08 Mon 05:52:18:621.028 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: OnMessage control in state: ClientInit 2026-06-08 Mon 05:52:18:621.064 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-06-08 Mon 05:52:18:621.072 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-06-08 Mon 05:52:18:621.087 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Received Ctrl Message with size 0 2026-06-08 Mon 05:52:18:621.324 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = ClientInit state_since = 1780897938615262 last_state = ClientInit last_event = scm::EvSandeshSend last_event_at = 1780897938615574 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 1 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 8, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 8, _iter59->first = sent _iter59->second = 1, ] ] max_sm_queue_count = 5 proxy seq 7 2026-06-08 Mon 05:52:18:621.357 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-06-08 Mon 05:52:18:621.379 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 6 2026-06-08 Mon 05:52:18:621.390 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-06-08 Mon 05:52:18:621.397 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-06-08 Mon 05:52:18:621.406 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-06-08 Mon 05:52:18:621.451 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Established 2026-06-08 Mon 05:52:23:615.135 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0f770 2026-06-08 Mon 05:52:23:615.298 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a0f770 2026-06-08 Mon 05:52:23:615.308 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:52:23:615.324 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:52:23:615.332 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10290 2026-06-08 Mon 05:52:23:615.371 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10e50 2026-06-08 Mon 05:52:23:615.405 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a11930 2026-06-08 Mon 05:52:23:615.475 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a10290 2026-06-08 Mon 05:52:23:615.482 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:52:23:615.490 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:52:23:615.497 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a10e50 2026-06-08 Mon 05:52:23:615.501 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:52:23:615.514 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:52:23:615.521 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a11930 2026-06-08 Mon 05:52:23:615.525 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:52:23:615.531 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:52:23:615.986 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:52:23:616.005 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:52:23:616.017 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:52:23:616.024 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:52:23:616.033 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:52:23:616.040 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:52:23:616.050 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:52:23:616.056 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:52:23:616.085 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:52:23:616.135 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:52:23:616.146 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:52:23:616.156 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:52:23:616.339 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12410 2026-06-08 Mon 05:52:23:616.425 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a12410 2026-06-08 Mon 05:52:23:616.451 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:52:23:616.465 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:52:23:617.355 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:52:23:617.391 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:52:23:617.422 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:52:23:619.362 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12ef0 2026-06-08 Mon 05:52:23:619.447 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a139d0 2026-06-08 Mon 05:52:23:619.482 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a144b0 2026-06-08 Mon 05:52:23:619.508 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a14f90 2026-06-08 Mon 05:52:23:619.534 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a15a70 2026-06-08 Mon 05:52:23:619.639 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a12ef0 2026-06-08 Mon 05:52:23:619.663 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:52:23:619.701 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:52:23:619.710 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a139d0 2026-06-08 Mon 05:52:23:619.714 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:52:23:619.721 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:52:23:619.736 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a144b0 2026-06-08 Mon 05:52:23:619.740 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:52:23:619.746 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:52:23:619.752 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a14f90 2026-06-08 Mon 05:52:23:619.756 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:52:23:619.761 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:52:23:619.768 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a15a70 2026-06-08 Mon 05:52:23:619.772 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:52:23:619.783 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:52:23:620.340 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:52:23:620.359 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:52:23:620.369 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:52:23:620.376 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:52:23:620.386 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:52:23:620.392 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:52:23:620.402 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:52:23:620.409 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:52:23:620.417 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:52:23:620.424 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:52:23:620.450 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:52:23:620.499 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:52:23:620.510 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:52:23:620.519 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:52:23:620.527 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:52:48:619.829 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-06-08 Mon 05:53:39:365.542 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Disconnect status: -1 0x2a12410 2026-06-08 Mon 05:53:39:365.623 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-08 Mon 05:53:39:365.785 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Disconnect status: -1 0x2a11930 2026-06-08 Mon 05:53:39:365.798 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-08 Mon 05:53:39:365.824 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Disconnect status: -1 0x2a10e50 2026-06-08 Mon 05:53:39:365.832 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-08 Mon 05:53:39:365.853 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Disconnect status: -1 0x2a10290 2026-06-08 Mon 05:53:39:365.861 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-08 Mon 05:53:39:365.884 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-06-08 Mon 05:53:39:365.890 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Disconnect status: -1 0x2a0f770 2026-06-08 Mon 05:53:39:365.896 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-08 Mon 05:53:39:366.079 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12410 2026-06-08 Mon 05:53:39:366.124 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a11930 2026-06-08 Mon 05:53:39:366.146 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10e50 2026-06-08 Mon 05:53:39:366.161 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10290 2026-06-08 Mon 05:53:39:366.183 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0f770 2026-06-08 Mon 05:53:39:421.212 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Disconnect status: -1 0x2a0e130 2026-06-08 Mon 05:53:39:421.260 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-08 Mon 05:53:39:421.297 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Disconnect status: -1 0x2a0ec50 2026-06-08 Mon 05:53:39:421.308 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-08 Mon 05:53:39:421.433 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Disconnect status: -1 0x2a0d830 2026-06-08 Mon 05:53:39:421.446 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-08 Mon 05:53:39:421.490 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-06-08 Mon 05:53:39:421.498 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Disconnect status: -1 0x2a08180 2026-06-08 Mon 05:53:39:421.507 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-08 Mon 05:53:39:421.526 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Disconnect status: -1 0x2a0d180 2026-06-08 Mon 05:53:39:421.536 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-08 Mon 05:53:39:421.683 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Disconnect status: -1 0x2a15a70 2026-06-08 Mon 05:53:39:421.695 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-08 Mon 05:53:39:421.721 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Disconnect status: -1 0x2a14f90 2026-06-08 Mon 05:53:39:421.734 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-08 Mon 05:53:39:421.758 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Disconnect status: -1 0x2a144b0 2026-06-08 Mon 05:53:39:421.769 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-08 Mon 05:53:39:421.788 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Disconnect status: -1 0x2a139d0 2026-06-08 Mon 05:53:39:421.797 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-08 Mon 05:53:39:421.817 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-06-08 Mon 05:53:39:421.823 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Disconnect status: -1 0x2a12ef0 2026-06-08 Mon 05:53:39:421.831 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-08 Mon 05:53:39:426.070 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0e130 2026-06-08 Mon 05:53:39:426.118 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0ec50 2026-06-08 Mon 05:53:39:426.132 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d830 2026-06-08 Mon 05:53:39:426.144 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a08180 2026-06-08 Mon 05:53:39:426.156 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d180 2026-06-08 Mon 05:53:39:426.178 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a15a70 2026-06-08 Mon 05:53:39:426.191 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a14f90 2026-06-08 Mon 05:53:39:426.202 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a144b0 2026-06-08 Mon 05:53:39:426.213 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a139d0 2026-06-08 Mon 05:53:39:426.224 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12ef0 2026-06-08 Mon 05:53:39:567.269 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.50:57232::10.0.0.38:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 506 2026-06-08 Mon 05:53:39:567.303 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.50:57232::10.0.0.38:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2026-06-08 Mon 05:53:39:567.322 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:57232::10.0.0.38:8086(-1) OnSessionEvent TCP Connection Closed 2026-06-08 Mon 05:53:39:567.328 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:57232::10.0.0.38:8086(-1):EvTcpClose 2026-06-08 Mon 05:53:39:567.367 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvTcpClose in state Established 2026-06-08 Mon 05:53:39:567.379 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Established : EvTcpClose 2026-06-08 Mon 05:53:39:567.602 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-06-08 Mon 05:53:39:567.674 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:53:39:568.599 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-06-08 Mon 05:53:39:569.954 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: -(-1):EvTcpConnectFail 2026-06-08 Mon 05:53:39:570.064 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SANDESH: Send FAILED: 1780898019567441 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = cn-jenkins-deploy-platform-ansible-os-5902-1. collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 106744 calls = 14 average_bytes = 7624 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 12, _iter59->first = sent _iter59->second = 17, ] ] ] 2026-06-08 Mon 05:53:39:570.095 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:53:39:570.107 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:53:39:570.131 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SANDESH: Send FAILED: 1780898019567490 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:53:39:570.139 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:53:39:570.150 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2026-06-08 Mon 05:53:39:570.211 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:53:39:570.301 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SANDESH: Send FAILED: 1780898019567660 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 106744 calls = 14 average_bytes = 7624 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 12, _iter59->first = sent _iter59->second = 17, ] ] ] 2026-06-08 Mon 05:53:39:570.315 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:53:39:570.324 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-06-08 Mon 05:53:39:570.330 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Connect : EvTcpConnectFail 2026-06-08 Mon 05:53:39:570.422 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2026-06-08 Mon 05:53:39:570.446 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Idle 2026-06-08 Mon 05:53:39:570.509 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:53:39:570.598 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SANDESH: Send FAILED: 1780898019570377 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 106744 calls = 14 average_bytes = 7624 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 15, _iter59->first = sent _iter59->second = 17, ] ] ] 2026-06-08 Mon 05:53:39:570.614 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:53:39:570.622 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:53:39:570.637 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SANDESH: Send FAILED: 1780898019570402 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:53:39:570.642 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:53:39:570.648 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-08 Mon 05:53:39:570.664 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:53:39:570.739 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SANDESH: Send FAILED: 1780898019570498 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 106744 calls = 14 average_bytes = 7624 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 15, _iter59->first = sent _iter59->second = 17, ] ] ] 2026-06-08 Mon 05:53:39:570.753 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:53:40:570.558 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-06-08 Mon 05:53:40:570.786 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-06-08 Mon 05:53:40:570.876 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:53:40:570.899 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SANDESH: Send FAILED: 1780898020570635 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:53:40:570.907 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:53:40:570.916 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:53:40:571.066 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SANDESH: Send FAILED: 1780898020570862 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 106744 calls = 14 average_bytes = 7624 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 17, ] ] ] 2026-06-08 Mon 05:53:40:571.096 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:53:40:572.938 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:39320::10.0.0.50:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 281 2026-06-08 Mon 05:53:40:572.957 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:39320::10.0.0.50:8086(-1) OnSessionEvent TCP Connected 2026-06-08 Mon 05:53:40:572.965 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:39320::10.0.0.50:8086(-1):EvTcpConnected 2026-06-08 Mon 05:53:40:573.057 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796886316608, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-06-08 Mon 05:53:40:573.082 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796886316608, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-06-08 Mon 05:53:40:573.117 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796886316608, Pid 1]: ClientInit 2026-06-08 Mon 05:53:40:573.136 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796886316608, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5902-2.:contrail-query-engine:0:Database count 2 2026-06-08 Mon 05:53:40:573.213 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796886316608, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:53:40:573.220 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796886316608, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-08 Mon 05:53:40:573.238 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796886316608, Pid 1]: SANDESH: Send FAILED: 1780898020573111 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:53:40:573.252 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796886316608, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-08 Mon 05:53:40:573.259 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796886316608, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:53:40:573.264 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796886316608, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-06-08 Mon 05:53:40:573.278 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796886316608, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:53:40:573.283 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796886316608, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-08 Mon 05:53:40:573.377 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796886316608, Pid 1]: SANDESH: Send FAILED: 1780898020573197 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 106744 calls = 14 average_bytes = 7624 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 17, ] ] ] 2026-06-08 Mon 05:53:40:573.392 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796886316608, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-08 Mon 05:53:40:580.242 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:39320::10.0.0.50:8086(-1) OnSessionEvent TCP Connection Closed 2026-06-08 Mon 05:53:40:580.265 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:39320::10.0.0.50:8086(-1):EvTcpClose 2026-06-08 Mon 05:53:40:580.301 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-06-08 Mon 05:53:40:580.317 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: ClientInit : EvTcpClose 2026-06-08 Mon 05:53:40:580.419 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Return to idle with 1000 2026-06-08 Mon 05:53:40:580.427 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Idle 2026-06-08 Mon 05:53:40:580.481 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:53:40:580.574 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SANDESH: Send FAILED: 1780898020580377 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 108317 calls = 15 average_bytes = 7221 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 22, _iter59->first = sent _iter59->second = 18, ] ] ] 2026-06-08 Mon 05:53:40:580.588 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:53:40:580.597 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:53:40:580.616 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SANDESH: Send FAILED: 1780898020580405 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:53:40:580.628 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:53:40:580.638 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-08 Mon 05:53:40:580.665 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:53:40:580.743 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SANDESH: Send FAILED: 1780898020580459 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 2 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 108317 calls = 15 average_bytes = 7221 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 22, _iter59->first = sent _iter59->second = 18, ] ] ] 2026-06-08 Mon 05:53:40:580.754 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:53:41:581.065 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-06-08 Mon 05:53:41:581.339 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-06-08 Mon 05:53:41:581.452 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:53:41:581.505 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SANDESH: Send FAILED: 1780898021581139 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:53:41:581.513 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:53:41:581.521 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:53:41:581.616 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SANDESH: Send FAILED: 1780898021581436 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 108317 calls = 15 average_bytes = 7221 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 25, _iter59->first = sent _iter59->second = 18, ] ] ] 2026-06-08 Mon 05:53:41:581.629 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:53:41:583.031 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:49088::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 281 2026-06-08 Mon 05:53:41:583.069 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:49088::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2026-06-08 Mon 05:53:41:583.076 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:49088::10.0.0.38:8086(-1):EvTcpConnected 2026-06-08 Mon 05:53:41:583.136 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-06-08 Mon 05:53:41:583.303 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-06-08 Mon 05:53:41:583.340 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: ClientInit 2026-06-08 Mon 05:53:41:583.360 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5902-2.:contrail-query-engine:0:Database count 3 2026-06-08 Mon 05:53:41:583.459 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:53:41:583.479 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-08 Mon 05:53:41:583.499 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: SANDESH: Send FAILED: 1780898021583333 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:53:41:583.506 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-08 Mon 05:53:41:583.514 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:53:41:583.519 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-06-08 Mon 05:53:41:583.539 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:53:41:583.545 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-08 Mon 05:53:41:583.669 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: SANDESH: Send FAILED: 1780898021583445 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 108317 calls = 15 average_bytes = 7221 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 27, _iter59->first = sent _iter59->second = 18, ] ] ] 2026-06-08 Mon 05:53:41:583.693 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-08 Mon 05:53:42:870.630 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:49088::10.0.0.38:8086(-1) OnSessionEvent TCP Connection Closed 2026-06-08 Mon 05:53:42:870.687 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:49088::10.0.0.38:8086(-1):EvTcpClose 2026-06-08 Mon 05:53:42:870.768 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-06-08 Mon 05:53:42:870.799 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: ClientInit : EvTcpClose 2026-06-08 Mon 05:53:42:870.925 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Return to idle with 1000 2026-06-08 Mon 05:53:42:870.934 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Idle 2026-06-08 Mon 05:53:42:870.982 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:53:42:871.097 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: SANDESH: Send FAILED: 1780898022870875 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 109890 calls = 16 average_bytes = 6868 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 29, _iter59->first = sent _iter59->second = 19, ] ] ] 2026-06-08 Mon 05:53:42:871.125 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:53:42:871.134 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:53:42:871.151 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: SANDESH: Send FAILED: 1780898022870906 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:53:42:871.157 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:53:42:871.164 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-08 Mon 05:53:42:871.208 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:53:42:871.286 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: SANDESH: Send FAILED: 1780898022870970 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 109890 calls = 16 average_bytes = 6868 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 29, _iter59->first = sent _iter59->second = 19, ] ] ] 2026-06-08 Mon 05:53:42:871.300 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:53:43:871.040 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-06-08 Mon 05:53:43:871.333 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-06-08 Mon 05:53:43:871.432 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:53:43:871.456 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: SANDESH: Send FAILED: 1780898023871126 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:53:43:871.463 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:53:43:871.484 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:53:43:871.583 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: SANDESH: Send FAILED: 1780898023871414 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 109890 calls = 16 average_bytes = 6868 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 32, _iter59->first = sent _iter59->second = 19, ] ] ] 2026-06-08 Mon 05:53:43:871.604 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:53:43:873.252 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:42002::10.0.0.33:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 281 2026-06-08 Mon 05:53:43:873.280 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:42002::10.0.0.33:8086(-1) OnSessionEvent TCP Connected 2026-06-08 Mon 05:53:43:873.287 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:42002::10.0.0.33:8086(-1):EvTcpConnected 2026-06-08 Mon 05:53:43:873.728 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796848531008, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-06-08 Mon 05:53:43:873.745 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796848531008, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-06-08 Mon 05:53:43:873.792 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796848531008, Pid 1]: ClientInit 2026-06-08 Mon 05:53:43:873.804 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796848531008, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5902-2.:contrail-query-engine:0:Database count 4 2026-06-08 Mon 05:53:43:873.878 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796848531008, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:53:43:873.886 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796848531008, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-08 Mon 05:53:43:873.909 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796848531008, Pid 1]: SANDESH: Send FAILED: 1780898023873783 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:53:43:873.927 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796848531008, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-08 Mon 05:53:43:873.935 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796848531008, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:53:43:873.940 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796848531008, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-06-08 Mon 05:53:43:873.960 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796848531008, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:53:43:873.965 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796848531008, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-08 Mon 05:53:43:874.061 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796848531008, Pid 1]: SANDESH: Send FAILED: 1780898023873865 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 109890 calls = 16 average_bytes = 6868 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 34, _iter59->first = sent _iter59->second = 19, ] ] ] 2026-06-08 Mon 05:53:43:874.076 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796848531008, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-08 Mon 05:53:43:880.038 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:42002::10.0.0.33:8086(-1) OnSessionEvent TCP Connection Closed 2026-06-08 Mon 05:53:43:880.060 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:42002::10.0.0.33:8086(-1):EvTcpClose 2026-06-08 Mon 05:53:43:880.107 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-06-08 Mon 05:53:43:880.117 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: ClientInit : EvTcpClose 2026-06-08 Mon 05:53:43:880.242 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Return to idle with 1000 2026-06-08 Mon 05:53:43:880.251 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Idle 2026-06-08 Mon 05:53:43:880.296 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:53:43:880.398 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: SANDESH: Send FAILED: 1780898023880192 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 111463 calls = 17 average_bytes = 6556 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 36, _iter59->first = sent _iter59->second = 20, ] ] ] 2026-06-08 Mon 05:53:43:880.417 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:53:43:880.426 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:53:43:880.444 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: SANDESH: Send FAILED: 1780898023880227 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:53:43:880.450 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:53:43:880.457 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-08 Mon 05:53:43:880.500 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:53:43:880.579 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: SANDESH: Send FAILED: 1780898023880285 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 111463 calls = 17 average_bytes = 6556 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 36, _iter59->first = sent _iter59->second = 20, ] ] ] 2026-06-08 Mon 05:53:43:880.590 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:53:44:366.158 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12410 2026-06-08 Mon 05:53:44:366.295 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12410 2026-06-08 Mon 05:53:44:366.324 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a11930 2026-06-08 Mon 05:53:44:366.362 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10e50 2026-06-08 Mon 05:53:44:366.390 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10290 2026-06-08 Mon 05:53:44:366.415 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0f770 2026-06-08 Mon 05:53:44:366.445 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a11930 2026-06-08 Mon 05:53:44:366.460 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10e50 2026-06-08 Mon 05:53:44:366.481 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10290 2026-06-08 Mon 05:53:44:366.492 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0f770 2026-06-08 Mon 05:53:44:426.154 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0e130 2026-06-08 Mon 05:53:44:426.318 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0ec50 2026-06-08 Mon 05:53:44:426.363 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d830 2026-06-08 Mon 05:53:44:426.400 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a08180 2026-06-08 Mon 05:53:44:426.436 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d180 2026-06-08 Mon 05:53:44:426.482 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a15a70 2026-06-08 Mon 05:53:44:426.528 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a14f90 2026-06-08 Mon 05:53:44:426.564 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a144b0 2026-06-08 Mon 05:53:44:426.595 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a139d0 2026-06-08 Mon 05:53:44:426.684 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12ef0 2026-06-08 Mon 05:53:44:426.732 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0e130 2026-06-08 Mon 05:53:44:426.775 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0ec50 2026-06-08 Mon 05:53:44:426.825 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d830 2026-06-08 Mon 05:53:44:426.851 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a08180 2026-06-08 Mon 05:53:44:426.893 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d180 2026-06-08 Mon 05:53:44:426.925 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a15a70 2026-06-08 Mon 05:53:44:426.942 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a14f90 2026-06-08 Mon 05:53:44:426.957 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a144b0 2026-06-08 Mon 05:53:44:426.973 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a139d0 2026-06-08 Mon 05:53:44:426.989 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12ef0 2026-06-08 Mon 05:53:44:880.738 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-06-08 Mon 05:53:44:881.058 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-06-08 Mon 05:53:44:881.166 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:53:44:881.208 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SANDESH: Send FAILED: 1780898024880837 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:53:44:881.218 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:53:44:881.228 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:53:44:881.356 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SANDESH: Send FAILED: 1780898024881149 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 111463 calls = 17 average_bytes = 6556 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 39, _iter59->first = sent _iter59->second = 20, ] ] ] 2026-06-08 Mon 05:53:44:881.385 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:53:44:883.632 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:39388::10.0.0.50:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 281 2026-06-08 Mon 05:53:44:883.668 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:39388::10.0.0.50:8086(-1) OnSessionEvent TCP Connected 2026-06-08 Mon 05:53:44:883.679 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:39388::10.0.0.50:8086(-1):EvTcpConnected 2026-06-08 Mon 05:53:44:883.745 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-06-08 Mon 05:53:44:883.775 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-06-08 Mon 05:53:44:883.818 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: ClientInit 2026-06-08 Mon 05:53:44:883.837 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5902-2.:contrail-query-engine:0:Database count 5 2026-06-08 Mon 05:53:44:883.929 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:53:44:883.938 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-08 Mon 05:53:44:883.960 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SANDESH: Send FAILED: 1780898024883810 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:53:44:883.967 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-08 Mon 05:53:44:883.976 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:53:44:883.983 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-06-08 Mon 05:53:44:884.008 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:53:44:884.015 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-08 Mon 05:53:44:884.139 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SANDESH: Send FAILED: 1780898024883914 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 111463 calls = 17 average_bytes = 6556 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 41, _iter59->first = sent _iter59->second = 20, ] ] ] 2026-06-08 Mon 05:53:44:884.155 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-08 Mon 05:53:48:622.279 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-06-08 Mon 05:53:48:622.333 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:53:48:622.342 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-08 Mon 05:53:48:622.401 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: SANDESH: Send FAILED: 1780898028622201 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 64 dequeues = 64 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 86 dequeues = 86 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = ClientInit state_since = 1780898024883817 last_state = ClientInit last_event = scm::EvSandeshSend last_event_at = 1780898024884007 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 1 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 5 ] 2026-06-08 Mon 05:53:48:622.423 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-08 Mon 05:53:48:622.431 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:53:48:622.436 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-08 Mon 05:53:48:622.532 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: SANDESH: Send FAILED: 1780898028622262 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 113036 calls = 18 average_bytes = 6279 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 43, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-06-08 Mon 05:53:48:622.549 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-08 Mon 05:53:49:366.361 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12410 2026-06-08 Mon 05:53:49:366.547 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12410 2026-06-08 Mon 05:53:49:366.573 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a11930 2026-06-08 Mon 05:53:49:366.606 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10e50 2026-06-08 Mon 05:53:49:366.634 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10290 2026-06-08 Mon 05:53:49:366.658 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0f770 2026-06-08 Mon 05:53:49:366.706 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a11930 2026-06-08 Mon 05:53:49:366.720 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10e50 2026-06-08 Mon 05:53:49:366.730 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10290 2026-06-08 Mon 05:53:49:366.741 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0f770 2026-06-08 Mon 05:53:49:426.813 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0e130 2026-06-08 Mon 05:53:49:426.997 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0ec50 2026-06-08 Mon 05:53:49:427.036 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d830 2026-06-08 Mon 05:53:49:427.064 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a08180 2026-06-08 Mon 05:53:49:427.090 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d180 2026-06-08 Mon 05:53:49:427.116 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a15a70 2026-06-08 Mon 05:53:49:427.140 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a14f90 2026-06-08 Mon 05:53:49:427.164 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a144b0 2026-06-08 Mon 05:53:49:427.194 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a139d0 2026-06-08 Mon 05:53:49:427.218 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12ef0 2026-06-08 Mon 05:53:49:427.285 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0e130 2026-06-08 Mon 05:53:49:427.326 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0ec50 2026-06-08 Mon 05:53:49:427.341 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d830 2026-06-08 Mon 05:53:49:427.356 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a08180 2026-06-08 Mon 05:53:49:427.371 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d180 2026-06-08 Mon 05:53:49:427.385 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a15a70 2026-06-08 Mon 05:53:49:427.401 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a14f90 2026-06-08 Mon 05:53:49:427.415 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a144b0 2026-06-08 Mon 05:53:49:427.462 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12ef0 2026-06-08 Mon 05:53:49:427.487 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a139d0 2026-06-08 Mon 05:53:51:301.233 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.50:39388::10.0.0.50:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 506 2026-06-08 Mon 05:53:51:301.269 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.50:39388::10.0.0.50:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2026-06-08 Mon 05:53:51:304.989 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:39388::10.0.0.50:8086(-1) OnSessionEvent TCP Connection Closed 2026-06-08 Mon 05:53:51:305.022 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:39388::10.0.0.50:8086(-1):EvTcpClose 2026-06-08 Mon 05:53:51:307.144 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-06-08 Mon 05:53:51:307.172 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: ClientInit : EvTcpClose 2026-06-08 Mon 05:53:51:307.310 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Return to idle with 1000 2026-06-08 Mon 05:53:51:307.320 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Idle 2026-06-08 Mon 05:53:51:307.369 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:53:51:307.516 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: SANDESH: Send FAILED: 1780898031307247 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 113036 calls = 18 average_bytes = 6279 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 45, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-06-08 Mon 05:53:51:307.542 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:53:51:307.551 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:53:51:307.569 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: SANDESH: Send FAILED: 1780898031307283 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:53:51:307.575 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:53:51:307.583 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-08 Mon 05:53:51:307.591 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:53:51:307.697 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: SANDESH: Send FAILED: 1780898031307358 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 113036 calls = 18 average_bytes = 6279 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 45, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-06-08 Mon 05:53:51:307.710 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:53:52:307.447 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-06-08 Mon 05:53:52:307.727 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-06-08 Mon 05:53:52:307.856 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:53:52:307.892 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: SANDESH: Send FAILED: 1780898032307540 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:53:52:307.903 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:53:52:307.915 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:53:52:307.955 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-06-08 Mon 05:53:52:307.980 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: -(-1):EvTcpConnectFail 2026-06-08 Mon 05:53:52:308.083 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: SANDESH: Send FAILED: 1780898032307835 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 113036 calls = 18 average_bytes = 6279 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 48, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-06-08 Mon 05:53:52:308.109 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:53:52:308.120 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-06-08 Mon 05:53:52:308.127 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Connect : EvTcpConnectFail 2026-06-08 Mon 05:53:52:308.214 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2026-06-08 Mon 05:53:52:308.239 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Idle 2026-06-08 Mon 05:53:52:308.281 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:53:52:308.365 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: SANDESH: Send FAILED: 1780898032308171 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 113036 calls = 18 average_bytes = 6279 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 50, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-06-08 Mon 05:53:52:308.381 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:53:52:308.389 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:53:52:308.403 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: SANDESH: Send FAILED: 1780898032308194 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:53:52:308.409 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:53:52:308.415 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-08 Mon 05:53:52:308.441 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:53:52:308.589 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: SANDESH: Send FAILED: 1780898032308272 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 113036 calls = 18 average_bytes = 6279 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 50, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-06-08 Mon 05:53:52:308.619 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:53:54:366.636 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12410 2026-06-08 Mon 05:53:54:366.838 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12410 2026-06-08 Mon 05:53:54:369.275 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a11930 2026-06-08 Mon 05:53:54:369.365 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10e50 2026-06-08 Mon 05:53:54:369.407 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10290 2026-06-08 Mon 05:53:54:369.438 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0f770 2026-06-08 Mon 05:53:54:369.488 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a11930 2026-06-08 Mon 05:53:54:369.506 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10e50 2026-06-08 Mon 05:53:54:369.517 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10290 2026-06-08 Mon 05:53:54:369.527 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0f770 2026-06-08 Mon 05:53:54:427.360 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0e130 2026-06-08 Mon 05:53:54:427.506 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0ec50 2026-06-08 Mon 05:53:54:427.554 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d830 2026-06-08 Mon 05:53:54:427.587 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a08180 2026-06-08 Mon 05:53:54:427.620 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d180 2026-06-08 Mon 05:53:54:427.654 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a15a70 2026-06-08 Mon 05:53:54:427.682 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a14f90 2026-06-08 Mon 05:53:54:427.707 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a144b0 2026-06-08 Mon 05:53:54:427.732 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12ef0 2026-06-08 Mon 05:53:54:427.757 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a139d0 2026-06-08 Mon 05:53:54:427.793 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0e130 2026-06-08 Mon 05:53:54:427.815 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0ec50 2026-06-08 Mon 05:53:54:427.827 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d830 2026-06-08 Mon 05:53:54:427.839 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d180 2026-06-08 Mon 05:53:54:427.849 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a08180 2026-06-08 Mon 05:53:54:427.872 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a15a70 2026-06-08 Mon 05:53:54:427.915 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a14f90 2026-06-08 Mon 05:53:54:427.927 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a144b0 2026-06-08 Mon 05:53:54:427.947 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12ef0 2026-06-08 Mon 05:53:54:427.959 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a139d0 2026-06-08 Mon 05:53:59:369.318 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12410 2026-06-08 Mon 05:53:59:369.515 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12410 2026-06-08 Mon 05:53:59:369.550 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a11930 2026-06-08 Mon 05:53:59:369.596 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10e50 2026-06-08 Mon 05:53:59:369.633 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10290 2026-06-08 Mon 05:53:59:369.670 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0f770 2026-06-08 Mon 05:53:59:369.713 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a11930 2026-06-08 Mon 05:53:59:369.741 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10e50 2026-06-08 Mon 05:53:59:369.756 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10290 2026-06-08 Mon 05:53:59:369.770 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0f770 2026-06-08 Mon 05:53:59:427.870 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0e130 2026-06-08 Mon 05:53:59:428.032 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0ec50 2026-06-08 Mon 05:53:59:428.073 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d830 2026-06-08 Mon 05:53:59:428.112 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d180 2026-06-08 Mon 05:53:59:428.145 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a08180 2026-06-08 Mon 05:53:59:428.178 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a15a70 2026-06-08 Mon 05:53:59:428.209 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a14f90 2026-06-08 Mon 05:53:59:428.241 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a144b0 2026-06-08 Mon 05:53:59:428.273 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12ef0 2026-06-08 Mon 05:53:59:428.303 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a139d0 2026-06-08 Mon 05:53:59:428.349 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0e130 2026-06-08 Mon 05:53:59:428.377 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0ec50 2026-06-08 Mon 05:53:59:428.392 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d830 2026-06-08 Mon 05:53:59:428.408 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d180 2026-06-08 Mon 05:53:59:428.422 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a08180 2026-06-08 Mon 05:53:59:428.436 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a15a70 2026-06-08 Mon 05:53:59:428.450 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a14f90 2026-06-08 Mon 05:53:59:428.559 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a144b0 2026-06-08 Mon 05:53:59:428.578 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12ef0 2026-06-08 Mon 05:53:59:428.591 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a139d0 2026-06-08 Mon 05:54:04:369.804 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12410 2026-06-08 Mon 05:54:04:369.982 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a11930 2026-06-08 Mon 05:54:04:370.017 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10e50 2026-06-08 Mon 05:54:04:370.061 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10290 2026-06-08 Mon 05:54:04:370.103 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12410 2026-06-08 Mon 05:54:04:370.128 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a11930 2026-06-08 Mon 05:54:04:370.140 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10e50 2026-06-08 Mon 05:54:04:370.150 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10290 2026-06-08 Mon 05:54:04:370.159 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0f770 2026-06-08 Mon 05:54:04:370.187 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0f770 2026-06-08 Mon 05:54:04:428.437 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0e130 2026-06-08 Mon 05:54:04:428.739 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0ec50 2026-06-08 Mon 05:54:04:428.803 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d830 2026-06-08 Mon 05:54:04:428.833 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d180 2026-06-08 Mon 05:54:04:428.859 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a08180 2026-06-08 Mon 05:54:04:428.927 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a15a70 2026-06-08 Mon 05:54:04:428.975 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a14f90 2026-06-08 Mon 05:54:04:428.996 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a144b0 2026-06-08 Mon 05:54:04:429.065 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12ef0 2026-06-08 Mon 05:54:04:429.094 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a139d0 2026-06-08 Mon 05:54:04:429.135 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0e130 2026-06-08 Mon 05:54:04:429.160 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0ec50 2026-06-08 Mon 05:54:04:429.172 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d830 2026-06-08 Mon 05:54:04:429.183 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d180 2026-06-08 Mon 05:54:04:429.208 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a08180 2026-06-08 Mon 05:54:04:429.226 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a15a70 2026-06-08 Mon 05:54:04:429.236 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a14f90 2026-06-08 Mon 05:54:04:429.301 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a144b0 2026-06-08 Mon 05:54:04:429.314 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12ef0 2026-06-08 Mon 05:54:04:429.339 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a139d0 2026-06-08 Mon 05:54:09:370.234 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12410 2026-06-08 Mon 05:54:09:370.423 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a11930 2026-06-08 Mon 05:54:09:370.461 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10e50 2026-06-08 Mon 05:54:09:370.519 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10290 2026-06-08 Mon 05:54:09:370.573 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12410 2026-06-08 Mon 05:54:09:370.606 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a11930 2026-06-08 Mon 05:54:09:370.624 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10e50 2026-06-08 Mon 05:54:09:370.640 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10290 2026-06-08 Mon 05:54:09:370.652 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0f770 2026-06-08 Mon 05:54:09:370.698 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0f770 2026-06-08 Mon 05:54:09:429.213 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0e130 2026-06-08 Mon 05:54:09:429.379 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0ec50 2026-06-08 Mon 05:54:09:429.423 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d830 2026-06-08 Mon 05:54:09:429.465 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d180 2026-06-08 Mon 05:54:09:429.509 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a08180 2026-06-08 Mon 05:54:09:429.543 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a15a70 2026-06-08 Mon 05:54:09:429.578 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a14f90 2026-06-08 Mon 05:54:09:429.612 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a144b0 2026-06-08 Mon 05:54:09:429.646 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12ef0 2026-06-08 Mon 05:54:09:429.719 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a139d0 2026-06-08 Mon 05:54:09:429.782 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0e130 2026-06-08 Mon 05:54:09:429.831 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0ec50 2026-06-08 Mon 05:54:09:429.858 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d830 2026-06-08 Mon 05:54:09:429.887 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a15a70 2026-06-08 Mon 05:54:09:429.940 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a14f90 2026-06-08 Mon 05:54:09:429.959 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a144b0 2026-06-08 Mon 05:54:09:429.976 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d180 2026-06-08 Mon 05:54:09:429.994 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a139d0 2026-06-08 Mon 05:54:09:430.010 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a08180 2026-06-08 Mon 05:54:09:430.028 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12ef0 2026-06-08 Mon 05:54:14:370.765 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12410 2026-06-08 Mon 05:54:14:370.906 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a11930 2026-06-08 Mon 05:54:14:370.939 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10e50 2026-06-08 Mon 05:54:14:370.964 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10290 2026-06-08 Mon 05:54:14:370.989 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0f770 2026-06-08 Mon 05:54:14:371.024 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12410 2026-06-08 Mon 05:54:14:371.049 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a11930 2026-06-08 Mon 05:54:14:371.060 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10e50 2026-06-08 Mon 05:54:14:371.070 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10290 2026-06-08 Mon 05:54:14:371.080 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0f770 2026-06-08 Mon 05:54:14:429.854 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0e130 2026-06-08 Mon 05:54:14:429.992 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0ec50 2026-06-08 Mon 05:54:14:430.031 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d830 2026-06-08 Mon 05:54:14:430.058 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a15a70 2026-06-08 Mon 05:54:14:430.085 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a14f90 2026-06-08 Mon 05:54:14:430.110 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a144b0 2026-06-08 Mon 05:54:14:430.133 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d180 2026-06-08 Mon 05:54:14:430.160 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a139d0 2026-06-08 Mon 05:54:14:430.186 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a08180 2026-06-08 Mon 05:54:14:430.244 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12ef0 2026-06-08 Mon 05:54:14:430.297 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0e130 2026-06-08 Mon 05:54:14:430.334 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0ec50 2026-06-08 Mon 05:54:14:430.352 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d830 2026-06-08 Mon 05:54:14:430.370 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d180 2026-06-08 Mon 05:54:14:430.388 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a15a70 2026-06-08 Mon 05:54:14:430.404 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a14f90 2026-06-08 Mon 05:54:14:430.415 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a144b0 2026-06-08 Mon 05:54:14:430.425 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a139d0 2026-06-08 Mon 05:54:14:430.435 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a08180 2026-06-08 Mon 05:54:14:430.446 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12ef0 2026-06-08 Mon 05:54:18:622.715 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797404309056, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:54:18:622.873 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797404309056, Pid 1]: SANDESH: Send FAILED: 1780898058622524 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 74 dequeues = 74 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 101 dequeues = 101 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1780898032308238 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1780898032308439 ] max_sm_queue_count = 5 ] 2026-06-08 Mon 05:54:18:622.896 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797404309056, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:54:18:622.908 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797404309056, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:54:18:623.065 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797404309056, Pid 1]: SANDESH: Send FAILED: 1780898058622644 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 113036 calls = 18 average_bytes = 6279 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 53, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-06-08 Mon 05:54:18:623.107 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797404309056, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:54:19:371.132 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12410 2026-06-08 Mon 05:54:19:371.479 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a11930 2026-06-08 Mon 05:54:19:371.564 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10e50 2026-06-08 Mon 05:54:19:371.593 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10290 2026-06-08 Mon 05:54:19:371.638 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12410 2026-06-08 Mon 05:54:19:371.667 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a11930 2026-06-08 Mon 05:54:19:371.678 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10e50 2026-06-08 Mon 05:54:19:371.689 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10290 2026-06-08 Mon 05:54:19:371.698 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0f770 2026-06-08 Mon 05:54:19:371.726 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0f770 2026-06-08 Mon 05:54:19:430.393 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0e130 2026-06-08 Mon 05:54:19:430.625 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0ec50 2026-06-08 Mon 05:54:19:430.702 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d830 2026-06-08 Mon 05:54:19:430.744 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d180 2026-06-08 Mon 05:54:19:430.782 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a15a70 2026-06-08 Mon 05:54:19:430.873 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a14f90 2026-06-08 Mon 05:54:19:430.929 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a144b0 2026-06-08 Mon 05:54:19:430.981 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a139d0 2026-06-08 Mon 05:54:19:431.026 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a08180 2026-06-08 Mon 05:54:19:431.074 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12ef0 2026-06-08 Mon 05:54:19:431.124 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0e130 2026-06-08 Mon 05:54:19:431.160 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0ec50 2026-06-08 Mon 05:54:19:431.179 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d830 2026-06-08 Mon 05:54:19:431.209 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d180 2026-06-08 Mon 05:54:19:431.239 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a15a70 2026-06-08 Mon 05:54:19:431.254 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a14f90 2026-06-08 Mon 05:54:19:431.267 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a144b0 2026-06-08 Mon 05:54:19:431.280 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a08180 2026-06-08 Mon 05:54:19:431.291 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12ef0 2026-06-08 Mon 05:54:19:434.259 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a139d0 2026-06-08 Mon 05:54:22:308.361 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-06-08 Mon 05:54:22:308.600 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-06-08 Mon 05:54:22:308.705 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:54:22:308.731 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: SANDESH: Send FAILED: 1780898062308442 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:54:22:308.746 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:54:22:308.753 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:54:22:308.804 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-06-08 Mon 05:54:22:308.893 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: SANDESH: Send FAILED: 1780898062308689 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 113036 calls = 18 average_bytes = 6279 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 55, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-06-08 Mon 05:54:22:308.919 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:54:22:308.895 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: -(-1):EvTcpConnectFail 2026-06-08 Mon 05:54:22:308.970 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-06-08 Mon 05:54:22:308.979 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Connect : EvTcpConnectFail 2026-06-08 Mon 05:54:22:309.088 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2026-06-08 Mon 05:54:22:309.126 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Idle 2026-06-08 Mon 05:54:22:309.186 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:54:22:309.282 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: SANDESH: Send FAILED: 1780898062309043 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 113036 calls = 18 average_bytes = 6279 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 57, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-06-08 Mon 05:54:22:309.296 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:54:22:309.304 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:54:22:309.319 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: SANDESH: Send FAILED: 1780898062309068 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:54:22:309.326 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:54:22:309.333 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-08 Mon 05:54:22:309.352 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:54:22:309.434 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: SANDESH: Send FAILED: 1780898062309175 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 113036 calls = 18 average_bytes = 6279 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 57, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-06-08 Mon 05:54:22:309.445 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:54:23:616.052 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796848531008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:54:23:616.136 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796848531008, Pid 1]: SANDESH: Send FAILED: 1780898063615993 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.111607 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2026-06-08 Mon 05:54:23:616.158 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796848531008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2026-06-08 Mon 05:54:24:371.727 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12410 2026-06-08 Mon 05:54:24:371.910 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a11930 2026-06-08 Mon 05:54:24:371.947 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10e50 2026-06-08 Mon 05:54:24:371.973 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10290 2026-06-08 Mon 05:54:24:372.010 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12410 2026-06-08 Mon 05:54:24:372.034 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a11930 2026-06-08 Mon 05:54:24:372.044 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10e50 2026-06-08 Mon 05:54:24:372.054 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10290 2026-06-08 Mon 05:54:24:372.072 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0f770 2026-06-08 Mon 05:54:24:372.105 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0f770 2026-06-08 Mon 05:54:24:431.190 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0e130 2026-06-08 Mon 05:54:24:431.305 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0ec50 2026-06-08 Mon 05:54:24:431.343 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d830 2026-06-08 Mon 05:54:24:431.372 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d180 2026-06-08 Mon 05:54:24:431.400 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a15a70 2026-06-08 Mon 05:54:24:431.429 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a14f90 2026-06-08 Mon 05:54:24:431.456 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a144b0 2026-06-08 Mon 05:54:24:431.499 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a08180 2026-06-08 Mon 05:54:24:431.531 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12ef0 2026-06-08 Mon 05:54:24:431.568 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0e130 2026-06-08 Mon 05:54:24:431.590 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0ec50 2026-06-08 Mon 05:54:24:431.602 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d830 2026-06-08 Mon 05:54:24:431.612 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d180 2026-06-08 Mon 05:54:24:431.623 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a15a70 2026-06-08 Mon 05:54:24:431.669 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a144b0 2026-06-08 Mon 05:54:24:431.682 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a14f90 2026-06-08 Mon 05:54:24:431.711 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a08180 2026-06-08 Mon 05:54:24:431.723 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12ef0 2026-06-08 Mon 05:54:24:434.356 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a139d0 2026-06-08 Mon 05:54:24:434.673 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a139d0 2026-06-08 Mon 05:54:29:372.103 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12410 2026-06-08 Mon 05:54:29:372.261 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a11930 2026-06-08 Mon 05:54:29:372.296 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10e50 2026-06-08 Mon 05:54:29:372.345 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12410 2026-06-08 Mon 05:54:29:372.368 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a11930 2026-06-08 Mon 05:54:29:372.379 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10e50 2026-06-08 Mon 05:54:29:372.388 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10290 2026-06-08 Mon 05:54:29:372.413 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0f770 2026-06-08 Mon 05:54:29:372.438 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10290 2026-06-08 Mon 05:54:29:372.450 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0f770 2026-06-08 Mon 05:54:29:431.657 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0e130 2026-06-08 Mon 05:54:29:431.792 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0ec50 2026-06-08 Mon 05:54:29:431.832 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d830 2026-06-08 Mon 05:54:29:431.862 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d180 2026-06-08 Mon 05:54:29:431.893 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a15a70 2026-06-08 Mon 05:54:29:431.923 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a144b0 2026-06-08 Mon 05:54:29:431.951 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a14f90 2026-06-08 Mon 05:54:29:431.978 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a08180 2026-06-08 Mon 05:54:29:432.006 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12ef0 2026-06-08 Mon 05:54:29:432.108 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0e130 2026-06-08 Mon 05:54:29:432.137 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0ec50 2026-06-08 Mon 05:54:29:432.155 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d830 2026-06-08 Mon 05:54:29:432.173 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d180 2026-06-08 Mon 05:54:29:432.188 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a15a70 2026-06-08 Mon 05:54:29:432.199 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a08180 2026-06-08 Mon 05:54:29:432.212 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a144b0 2026-06-08 Mon 05:54:29:432.243 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a14f90 2026-06-08 Mon 05:54:29:432.260 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12ef0 2026-06-08 Mon 05:54:29:434.715 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a139d0 2026-06-08 Mon 05:54:29:435.051 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a139d0 2026-06-08 Mon 05:54:34:372.442 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12410 2026-06-08 Mon 05:54:34:413.984 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a11930 2026-06-08 Mon 05:54:34:414.058 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10e50 2026-06-08 Mon 05:54:34:414.118 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12410 2026-06-08 Mon 05:54:34:414.151 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a11930 2026-06-08 Mon 05:54:34:414.168 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10e50 2026-06-08 Mon 05:54:34:414.181 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10290 2026-06-08 Mon 05:54:34:414.218 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0f770 2026-06-08 Mon 05:54:34:414.257 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10290 2026-06-08 Mon 05:54:34:414.274 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0f770 2026-06-08 Mon 05:54:34:432.204 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0e130 2026-06-08 Mon 05:54:34:456.305 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0ec50 2026-06-08 Mon 05:54:34:456.354 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d830 2026-06-08 Mon 05:54:34:456.400 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d180 2026-06-08 Mon 05:54:34:456.436 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a15a70 2026-06-08 Mon 05:54:34:456.476 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a08180 2026-06-08 Mon 05:54:34:456.513 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a144b0 2026-06-08 Mon 05:54:34:456.539 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a14f90 2026-06-08 Mon 05:54:34:456.566 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12ef0 2026-06-08 Mon 05:54:34:456.590 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a139d0 2026-06-08 Mon 05:54:34:456.637 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0e130 2026-06-08 Mon 05:54:34:456.663 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0ec50 2026-06-08 Mon 05:54:34:456.675 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d830 2026-06-08 Mon 05:54:34:456.686 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a08180 2026-06-08 Mon 05:54:34:456.739 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a15a70 2026-06-08 Mon 05:54:34:456.751 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a144b0 2026-06-08 Mon 05:54:34:456.762 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a14f90 2026-06-08 Mon 05:54:34:456.772 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12ef0 2026-06-08 Mon 05:54:34:456.783 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a139d0 2026-06-08 Mon 05:54:34:459.399 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d180 2026-06-08 Mon 05:54:39:414.198 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12410 2026-06-08 Mon 05:54:39:414.344 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a11930 2026-06-08 Mon 05:54:39:414.391 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12410 2026-06-08 Mon 05:54:39:414.413 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a11930 2026-06-08 Mon 05:54:39:414.422 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10e50 2026-06-08 Mon 05:54:39:414.447 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10290 2026-06-08 Mon 05:54:39:414.470 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0f770 2026-06-08 Mon 05:54:39:414.511 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10e50 2026-06-08 Mon 05:54:39:414.525 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10290 2026-06-08 Mon 05:54:39:414.535 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0f770 2026-06-08 Mon 05:54:39:456.713 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0e130 2026-06-08 Mon 05:54:39:456.870 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0ec50 2026-06-08 Mon 05:54:39:456.901 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d830 2026-06-08 Mon 05:54:39:456.930 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a08180 2026-06-08 Mon 05:54:39:456.956 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a15a70 2026-06-08 Mon 05:54:39:456.983 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a144b0 2026-06-08 Mon 05:54:39:457.038 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a14f90 2026-06-08 Mon 05:54:39:457.079 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12ef0 2026-06-08 Mon 05:54:39:457.113 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a139d0 2026-06-08 Mon 05:54:39:457.167 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0e130 2026-06-08 Mon 05:54:39:457.200 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0ec50 2026-06-08 Mon 05:54:39:457.213 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d830 2026-06-08 Mon 05:54:39:457.229 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a08180 2026-06-08 Mon 05:54:39:457.246 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a15a70 2026-06-08 Mon 05:54:39:457.263 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a144b0 2026-06-08 Mon 05:54:39:457.278 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a14f90 2026-06-08 Mon 05:54:39:457.326 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12ef0 2026-06-08 Mon 05:54:39:457.342 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a139d0 2026-06-08 Mon 05:54:39:459.458 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d180 2026-06-08 Mon 05:54:39:459.798 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d180 2026-06-08 Mon 05:54:44:414.453 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12410 2026-06-08 Mon 05:54:44:414.605 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a11930 2026-06-08 Mon 05:54:44:414.651 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12410 2026-06-08 Mon 05:54:44:414.671 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a11930 2026-06-08 Mon 05:54:44:414.681 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10e50 2026-06-08 Mon 05:54:44:414.706 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10290 2026-06-08 Mon 05:54:44:414.743 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0f770 2026-06-08 Mon 05:54:44:414.783 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10e50 2026-06-08 Mon 05:54:44:414.796 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10290 2026-06-08 Mon 05:54:44:414.807 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0f770 2026-06-08 Mon 05:54:44:457.227 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0e130 2026-06-08 Mon 05:54:44:457.343 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0ec50 2026-06-08 Mon 05:54:44:457.408 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d830 2026-06-08 Mon 05:54:44:457.439 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a08180 2026-06-08 Mon 05:54:44:457.466 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a15a70 2026-06-08 Mon 05:54:44:457.502 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a144b0 2026-06-08 Mon 05:54:44:457.527 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a14f90 2026-06-08 Mon 05:54:44:457.576 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0e130 2026-06-08 Mon 05:54:44:457.596 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12ef0 2026-06-08 Mon 05:54:44:457.622 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a139d0 2026-06-08 Mon 05:54:44:457.653 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0ec50 2026-06-08 Mon 05:54:44:457.666 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d830 2026-06-08 Mon 05:54:44:457.677 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a08180 2026-06-08 Mon 05:54:44:457.758 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a15a70 2026-06-08 Mon 05:54:44:457.770 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a144b0 2026-06-08 Mon 05:54:44:457.781 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a14f90 2026-06-08 Mon 05:54:44:457.840 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12ef0 2026-06-08 Mon 05:54:44:457.851 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a139d0 2026-06-08 Mon 05:54:44:459.868 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d180 2026-06-08 Mon 05:54:44:460.173 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d180 2026-06-08 Mon 05:54:48:622.929 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796848531008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-06-08 Mon 05:54:48:623.011 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796861126208, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:54:48:623.097 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796861126208, Pid 1]: SANDESH: Send FAILED: 1780898088622823 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 82 dequeues = 82 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 112 dequeues = 112 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1780898062309125 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1780898063616037 ] max_sm_queue_count = 5 ] 2026-06-08 Mon 05:54:48:623.119 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796861126208, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:54:48:623.129 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796861126208, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:54:48:623.253 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796861126208, Pid 1]: SANDESH: Send FAILED: 1780898088622910 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 113036 calls = 18 average_bytes = 6279 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 61, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-06-08 Mon 05:54:48:623.269 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796861126208, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:54:49:414.738 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12410 2026-06-08 Mon 05:54:49:414.890 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a11930 2026-06-08 Mon 05:54:49:414.937 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12410 2026-06-08 Mon 05:54:49:414.960 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a11930 2026-06-08 Mon 05:54:49:414.969 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10e50 2026-06-08 Mon 05:54:49:414.994 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10290 2026-06-08 Mon 05:54:49:415.016 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0f770 2026-06-08 Mon 05:54:49:415.045 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10e50 2026-06-08 Mon 05:54:49:415.057 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10290 2026-06-08 Mon 05:54:49:415.067 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0f770 2026-06-08 Mon 05:54:49:457.659 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0e130 2026-06-08 Mon 05:54:49:457.831 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0ec50 2026-06-08 Mon 05:54:49:457.891 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d830 2026-06-08 Mon 05:54:49:457.932 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a08180 2026-06-08 Mon 05:54:49:457.971 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a15a70 2026-06-08 Mon 05:54:49:458.011 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a144b0 2026-06-08 Mon 05:54:49:458.047 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a14f90 2026-06-08 Mon 05:54:49:458.103 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0e130 2026-06-08 Mon 05:54:49:458.133 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0ec50 2026-06-08 Mon 05:54:49:458.151 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d830 2026-06-08 Mon 05:54:49:458.165 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12ef0 2026-06-08 Mon 05:54:49:458.204 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a139d0 2026-06-08 Mon 05:54:49:458.248 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a08180 2026-06-08 Mon 05:54:49:458.282 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a15a70 2026-06-08 Mon 05:54:49:458.299 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a144b0 2026-06-08 Mon 05:54:49:458.317 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a14f90 2026-06-08 Mon 05:54:49:458.380 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12ef0 2026-06-08 Mon 05:54:49:458.400 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a139d0 2026-06-08 Mon 05:54:49:460.220 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d180 2026-06-08 Mon 05:54:49:460.560 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d180 2026-06-08 Mon 05:54:52:309.595 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-06-08 Mon 05:54:52:310.100 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-06-08 Mon 05:54:52:310.118 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-06-08 Mon 05:54:52:310.142 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: -(-1):EvTcpConnectFail 2026-06-08 Mon 05:54:52:310.176 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:54:52:310.197 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: SANDESH: Send FAILED: 1780898092309983 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:54:52:310.204 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:54:52:310.212 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-06-08 Mon 05:54:52:310.218 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Connect : EvTcpConnectFail 2026-06-08 Mon 05:54:52:310.283 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2026-06-08 Mon 05:54:52:310.305 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Idle 2026-06-08 Mon 05:54:52:310.347 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:54:52:310.452 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: SANDESH: Send FAILED: 1780898092310164 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 113036 calls = 18 average_bytes = 6279 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 63, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-06-08 Mon 05:54:52:310.471 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:54:52:310.479 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:54:52:310.566 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: SANDESH: Send FAILED: 1780898092310249 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 113036 calls = 18 average_bytes = 6279 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 64, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-06-08 Mon 05:54:52:310.580 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:54:52:310.588 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:54:52:310.602 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: SANDESH: Send FAILED: 1780898092310266 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:54:52:310.607 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:54:52:310.614 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-08 Mon 05:54:52:310.633 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:54:52:310.709 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: SANDESH: Send FAILED: 1780898092310337 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 113036 calls = 18 average_bytes = 6279 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 93, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 64, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-06-08 Mon 05:54:52:310.723 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:54:54:415.070 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12410 2026-06-08 Mon 05:54:54:415.266 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a11930 2026-06-08 Mon 05:54:54:415.324 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12410 2026-06-08 Mon 05:54:54:415.349 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a11930 2026-06-08 Mon 05:54:54:415.358 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10e50 2026-06-08 Mon 05:54:54:415.383 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10290 2026-06-08 Mon 05:54:54:415.413 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0f770 2026-06-08 Mon 05:54:54:415.454 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10e50 2026-06-08 Mon 05:54:54:415.471 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a10290 2026-06-08 Mon 05:54:54:415.486 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0f770 2026-06-08 Mon 05:54:54:458.178 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0e130 2026-06-08 Mon 05:54:54:458.312 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0ec50 2026-06-08 Mon 05:54:54:458.365 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d830 2026-06-08 Mon 05:54:54:458.401 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a08180 2026-06-08 Mon 05:54:54:458.432 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a15a70 2026-06-08 Mon 05:54:54:458.460 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a144b0 2026-06-08 Mon 05:54:54:458.489 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a14f90 2026-06-08 Mon 05:54:54:458.570 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12ef0 2026-06-08 Mon 05:54:54:458.599 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a139d0 2026-06-08 Mon 05:54:54:458.636 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0e130 2026-06-08 Mon 05:54:54:458.662 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d830 2026-06-08 Mon 05:54:54:458.673 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0ec50 2026-06-08 Mon 05:54:54:458.684 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a08180 2026-06-08 Mon 05:54:54:458.695 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a15a70 2026-06-08 Mon 05:54:54:458.706 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a144b0 2026-06-08 Mon 05:54:54:458.753 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a14f90 2026-06-08 Mon 05:54:54:458.765 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a12ef0 2026-06-08 Mon 05:54:54:458.776 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a139d0 2026-06-08 Mon 05:54:54:460.638 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d180 2026-06-08 Mon 05:54:54:460.969 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: -1 0x2a0d180 2026-06-08 Mon 05:54:59:415.420 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12410 2026-06-08 Mon 05:54:59:415.588 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a11930 2026-06-08 Mon 05:54:59:415.641 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a12410 2026-06-08 Mon 05:54:59:415.651 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:54:59:415.674 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:54:59:415.687 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a11930 2026-06-08 Mon 05:54:59:415.693 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:54:59:415.703 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:54:59:415.710 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10e50 2026-06-08 Mon 05:54:59:415.754 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a10290 2026-06-08 Mon 05:54:59:415.790 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0f770 2026-06-08 Mon 05:54:59:415.861 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a10e50 2026-06-08 Mon 05:54:59:415.877 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:54:59:415.885 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:54:59:415.892 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a10290 2026-06-08 Mon 05:54:59:415.896 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:54:59:415.902 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:54:59:415.908 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a0f770 2026-06-08 Mon 05:54:59:415.912 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:54:59:415.918 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:54:59:415.942 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:54:59:415.950 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:54:59:415.960 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:54:59:415.967 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:54:59:416.015 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:54:59:416.080 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:54:59:416.093 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:54:59:416.100 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:54:59:416.113 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:54:59:416.117 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:54:59:416.178 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: SANDESH: Send FAILED: 1780898099416050 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-08 Mon 05:54:59:416.200 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:54:59:416.123 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:54:59:416.651 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:54:59:416.671 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:54:59:416.678 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:54:59:416.695 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:54:59:416.707 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:54:59:458.714 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0e130 2026-06-08 Mon 05:54:59:458.832 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d830 2026-06-08 Mon 05:54:59:458.871 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0ec50 2026-06-08 Mon 05:54:59:458.912 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a08180 2026-06-08 Mon 05:54:59:458.946 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a15a70 2026-06-08 Mon 05:54:59:458.979 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a144b0 2026-06-08 Mon 05:54:59:459.049 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a14f90 2026-06-08 Mon 05:54:59:459.117 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a12ef0 2026-06-08 Mon 05:54:59:459.150 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a139d0 2026-06-08 Mon 05:54:59:459.217 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a0e130 2026-06-08 Mon 05:54:59:459.236 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:54:59:459.276 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:54:59:459.287 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a0d830 2026-06-08 Mon 05:54:59:459.292 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:54:59:459.299 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:54:59:459.306 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a08180 2026-06-08 Mon 05:54:59:459.310 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:54:59:459.317 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:54:59:459.324 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a0ec50 2026-06-08 Mon 05:54:59:459.328 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:54:59:459.347 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:54:59:459.356 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a15a70 2026-06-08 Mon 05:54:59:459.361 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:54:59:459.367 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:54:59:459.374 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a144b0 2026-06-08 Mon 05:54:59:459.378 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:54:59:459.384 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:54:59:459.410 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a14f90 2026-06-08 Mon 05:54:59:459.414 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:54:59:459.421 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:54:59:459.427 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a12ef0 2026-06-08 Mon 05:54:59:459.431 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:54:59:459.438 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:54:59:459.468 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a139d0 2026-06-08 Mon 05:54:59:459.472 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:54:59:459.487 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:54:59:459.742 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:54:59:459.772 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:54:59:459.788 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:54:59:459.796 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:54:59:459.820 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:54:59:459.876 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:54:59:459.889 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:54:59:459.896 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:54:59:459.906 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:54:59:459.913 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:54:59:459.922 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:54:59:459.923 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797400110656, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:54:59:459.974 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797400110656, Pid 1]: SANDESH: Send FAILED: 1780898099459846 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-08 Mon 05:54:59:459.983 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797400110656, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:54:59:460.114 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:54:59:460.132 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:54:59:460.139 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:54:59:460.148 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:54:59:460.155 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:54:59:460.164 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:54:59:460.171 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:54:59:460.183 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:54:59:460.190 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:54:59:460.204 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:54:59:460.214 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:54:59:460.246 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:54:59:460.257 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:54:59:460.265 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:54:59:460.268 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:54:59:460.274 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:54:59:460.307 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:54:59:460.310 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: SANDESH: Send FAILED: 1780898099460229 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-08 Mon 05:54:59:460.330 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:54:59:461.005 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2a0d180 2026-06-08 Mon 05:54:59:461.246 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: RAC_Connect status: 0 0x2a0d180 2026-06-08 Mon 05:54:59:461.255 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:54:59:461.265 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-08 Mon 05:54:59:461.509 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-08 Mon 05:54:59:461.518 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-08 Mon 05:54:59:461.531 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-08 Mon 05:55:18:623.202 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796856927808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:55:18:623.272 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796856927808, Pid 1]: SANDESH: Send FAILED: 1780898118623078 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 92 dequeues = 92 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 125 dequeues = 125 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1780898092310304 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1780898099460261 ] max_sm_queue_count = 5 ] 2026-06-08 Mon 05:55:18:623.285 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796856927808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:55:18:623.299 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796856927808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:55:18:623.400 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796856927808, Pid 1]: SANDESH: Send FAILED: 1780898118623171 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 113036 calls = 18 average_bytes = 6279 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 153, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 71, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-06-08 Mon 05:55:18:623.416 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796856927808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:55:22:311.578 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796861126208, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-06-08 Mon 05:55:22:311.808 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796861126208, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-06-08 Mon 05:55:22:311.896 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796861126208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:55:22:311.919 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796861126208, Pid 1]: SANDESH: Send FAILED: 1780898122311662 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-08 Mon 05:55:22:311.932 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796861126208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:55:22:311.939 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796861126208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:55:22:312.036 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796861126208, Pid 1]: SANDESH: Send FAILED: 1780898122311879 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 113036 calls = 18 average_bytes = 6279 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 153, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 73, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-06-08 Mon 05:55:22:312.255 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796861126208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:55:22:319.224 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:43322::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 281 2026-06-08 Mon 05:55:22:319.256 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:43322::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2026-06-08 Mon 05:55:22:319.264 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797415649792, Pid 1]: 10.0.0.50:43322::10.0.0.38:8086(-1):EvTcpConnected 2026-06-08 Mon 05:55:22:321.743 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-06-08 Mon 05:55:22:321.805 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-06-08 Mon 05:55:22:321.950 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: ClientInit 2026-06-08 Mon 05:55:22:321.965 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5902-2.:contrail-query-engine:0:Database count 6 2026-06-08 Mon 05:55:22:322.058 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:55:22:322.068 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-08 Mon 05:55:22:322.085 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: SANDESH: Send FAILED: 1780898122321859 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-08 Mon 05:55:22:322.112 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-08 Mon 05:55:22:322.121 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:55:22:322.126 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-06-08 Mon 05:55:22:322.135 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:55:22:322.140 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-08 Mon 05:55:22:322.271 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: SANDESH: Send FAILED: 1780898122322045 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 113036 calls = 18 average_bytes = 6279 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 154, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 75, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-06-08 Mon 05:55:22:322.292 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139797408507456, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-08 Mon 05:55:22:512.354 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: OnMessage control in state: ClientInit 2026-06-08 Mon 05:55:22:512.415 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-06-08 Mon 05:55:22:512.430 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-06-08 Mon 05:55:22:512.449 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Received Ctrl Message with size 0 2026-06-08 Mon 05:55:22:512.638 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 113036 calls = 18 average_bytes = 6279 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 92 dequeues = 92 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 125 dequeues = 125 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1780898092310304 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1780898099460261 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 1 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 154, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 75, _iter59->first = sent _iter59->second = 21, ] ] max_sm_queue_count = 5 proxy seq 51 2026-06-08 Mon 05:55:22:512.677 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-06-08 Mon 05:55:22:512.695 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 38 2026-06-08 Mon 05:55:22:512.707 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-06-08 Mon 05:55:22:512.734 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5902-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.111607 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 3 2026-06-08 Mon 05:55:22:512.743 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2026-06-08 Mon 05:55:22:512.752 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-06-08 Mon 05:55:22:512.782 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: Established 2026-06-08 Mon 05:55:28:119.966 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: OnMessage control in state: Established 2026-06-08 Mon 05:55:28:120.042 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2026-06-08 Mon 05:55:28:120.051 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Established : EvSandeshMessageRecv 2026-06-08 Mon 05:55:28:120.086 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: Received Ctrl Message with size 3 2026-06-08 Mon 05:55:28:120.098 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 52, total = 0 2026-06-08 Mon 05:55:28:120.108 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 40, total = 0 2026-06-08 Mon 05:55:28:120.117 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 4, total = 0 2026-06-08 Mon 05:55:28:120.124 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796852729408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-06-08 Mon 05:55:48:623.698 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796877919808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-06-08 Mon 05:56:48:624.375 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796882118208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-06-08 Mon 05:57:48:625.047 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796890515008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2026-06-08 Mon 05:58:48:627.338 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796869523008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2026-06-08 Mon 05:58:51:181.102 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Processing scm::EvStop in state Established 2026-06-08 Mon 05:58:51:181.194 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Idle 2026-06-08 Mon 05:58:51:181.286 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-08 Mon 05:58:51:181.296 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-08 Mon 05:58:51:181.412 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: SANDESH: Send FAILED: 1780898331181274 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780897898560890 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 335432 calls = 40 average_bytes = 8385 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 22 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 154, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 77, _iter59->first = sent _iter59->second = 46, ] ] ] 2026-06-08 Mon 05:58:51:181.437 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 139796844332608, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:59:16:646.060 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: http-server-port 8091 2026-06-08 Mon 05:59:16:646.128 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Max-tasks 16 2026-06-08 Mon 05:59:16:646.135 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Max-slice 100 2026-06-08 Mon 05:59:16:646.140 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Collectors 10.0.0.38:8086 2026-06-08 Mon 05:59:16:646.144 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Collectors 10.0.0.50:8086 2026-06-08 Mon 05:59:16:646.148 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Collectors 10.0.0.33:8086 2026-06-08 Mon 05:59:16:646.167 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: SANDESH: ROLE : Generator 2026-06-08 Mon 05:59:16:646.172 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-06-08 Mon 05:59:16:646.176 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5902-2. 2026-06-08 Mon 05:59:16:646.180 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: SANDESH: NODE TYPE : Database 2026-06-08 Mon 05:59:16:646.184 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: SANDESH: INSTANCE ID : 0 2026-06-08 Mon 05:59:16:646.189 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-06-08 Mon 05:59:16:646.193 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-06-08 Mon 05:59:16:648.914 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: HTTP Introspect Init 2026-06-08 Mon 05:59:16:649.026 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 138 2026-06-08 Mon 05:59:16:649.108 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Sandesh Http Server Port: 8091 2026-06-08 Mon 05:59:16:649.159 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-06-08 Mon 05:59:16:649.166 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-06-08 Mon 05:59:16:649.474 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Idle 2026-06-08 Mon 05:59:16:649.614 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: SANDESH: No Client: 1780898356649551 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780898356649485 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-08 Mon 05:59:16:657.362 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-06-08 Mon 05:59:16:657.389 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-06-08 Mon 05:59:16:657.406 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Cassandra Servers: 10.0.0.38:9042 10.0.0.50:9042 10.0.0.33:9042 2026-06-08 Mon 05:59:16:661.197 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Processing scm::EvStart in state Idle 2026-06-08 Mon 05:59:16:667.075 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Disconnect 2026-06-08 Mon 05:59:16:667.309 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-06-08 Mon 05:59:16:667.543 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-06-08 Mon 05:59:16:667.632 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:59:16:667.653 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: SANDESH: Send FAILED: 1780898356667057 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-06-08 Mon 05:59:16:667.661 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:59:16:667.669 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:59:16:667.730 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: SANDESH: Send FAILED: 1780898356667292 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780898356649485 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-08 Mon 05:59:16:667.740 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:59:16:667.752 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:59:16:667.796 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: SANDESH: Send FAILED: 1780898356667343 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780898356649485 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-08 Mon 05:59:16:667.803 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:59:16:667.810 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:59:16:667.821 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: SANDESH: Send FAILED: 1780898356667371 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-06-08 Mon 05:59:16:667.831 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:59:16:667.837 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:59:16:667.875 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: SANDESH: Send FAILED: 1780898356667616 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780898356649485 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-08 Mon 05:59:16:667.883 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-08 Mon 05:59:16:753.077 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-06-08 Mon 05:59:16:755.280 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: RAC_Connect status: 0 0x14404f0 2026-06-08 Mon 05:59:16:755.299 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:59:16:755.316 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-08 Mon 05:59:16:755.327 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: RAC_Connect status: 0 0x14410b0 2026-06-08 Mon 05:59:16:755.331 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:59:16:755.337 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-08 Mon 05:59:16:755.345 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: RAC_Connect status: 0 0x1438580 2026-06-08 Mon 05:59:16:755.349 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:59:16:755.355 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-08 Mon 05:59:16:755.363 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: RAC_Connect status: 0 0x1441cf0 2026-06-08 Mon 05:59:16:755.367 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:59:16:755.373 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-08 Mon 05:59:16:755.381 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: RAC_Connect status: 0 0x143eb70 2026-06-08 Mon 05:59:16:755.385 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:59:16:755.391 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-08 Mon 05:59:16:755.398 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: RAC_Connect status: 0 0x143f830 2026-06-08 Mon 05:59:16:755.402 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:59:16:755.408 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-08 Mon 05:59:16:755.415 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: RAC_Connect status: 0 0x143d540 2026-06-08 Mon 05:59:16:755.419 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:59:16:755.425 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-08 Mon 05:59:16:755.433 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: RAC_Connect status: 0 0x143deb0 2026-06-08 Mon 05:59:16:755.439 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:59:16:755.448 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-08 Mon 05:59:16:755.460 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: RAC_Connect status: 0 0x1442930 2026-06-08 Mon 05:59:16:755.468 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:59:16:755.475 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-08 Mon 05:59:16:755.482 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: RAC_Connect status: 0 0x1443570 2026-06-08 Mon 05:59:16:755.486 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:59:16:755.492 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-08 Mon 05:59:16:755.499 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: RAC_Connect status: 0 0x14441b0 2026-06-08 Mon 05:59:16:755.503 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:59:16:755.509 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-08 Mon 05:59:16:755.516 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: RAC_Connect status: 0 0x1445a30 2026-06-08 Mon 05:59:16:755.520 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:59:16:755.526 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-08 Mon 05:59:16:755.533 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: RAC_Connect status: 0 0x1444df0 2026-06-08 Mon 05:59:16:755.537 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:59:16:755.547 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-08 Mon 05:59:16:755.557 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: RAC_Connect status: 0 0x1446670 2026-06-08 Mon 05:59:16:755.562 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:59:16:755.572 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-08 Mon 05:59:16:755.583 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: RAC_Connect status: 0 0x14472b0 2026-06-08 Mon 05:59:16:755.589 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: Connected to REDIS... 2026-06-08 Mon 05:59:16:755.607 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-08 Mon 05:59:16:755.631 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-08 Mon 05:59:16:755.679 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: SANDESH: Send FAILED: 1780898356755193 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-06-08 Mon 05:59:16:755.693 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-08 Mon 05:59:16:756.998 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:48730::10.0.0.50:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2026-06-08 Mon 05:59:16:757.028 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: 10.0.0.50:48730::10.0.0.50:8086(-1) OnSessionEvent TCP Connected 2026-06-08 Mon 05:59:16:757.036 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: 10.0.0.50:48730::10.0.0.50:8086(-1):EvTcpConnected 2026-06-08 Mon 05:59:16:757.272 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-08 Mon 05:59:16:757.283 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-08 Mon 05:59:16:757.286 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-06-08 Mon 05:59:16:757.312 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-06-08 Mon 05:59:16:757.356 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: ClientInit 2026-06-08 Mon 05:59:16:757.377 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5902-2.:contrail-query-engine:0:Database count 1 2026-06-08 Mon 05:59:16:757.446 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:59:16:757.454 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-08 Mon 05:59:16:757.469 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: SANDESH: Send FAILED: 1780898356757349 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-06-08 Mon 05:59:16:757.475 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-08 Mon 05:59:16:757.481 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:59:16:757.487 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-06-08 Mon 05:59:16:757.506 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:59:16:757.517 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-08 Mon 05:59:16:757.626 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: SANDESH: Send FAILED: 1780898356757436 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780898356649485 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 21, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-08 Mon 05:59:16:757.639 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-08 Mon 05:59:16:757.295 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-08 Mon 05:59:16:757.845 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-08 Mon 05:59:16:757.864 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-08 Mon 05:59:16:757.871 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-08 Mon 05:59:16:757.883 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-08 Mon 05:59:16:757.889 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-08 Mon 05:59:16:757.899 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-08 Mon 05:59:16:757.906 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-08 Mon 05:59:16:758.086 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:59:16:758.095 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-08 Mon 05:59:16:758.114 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: SANDESH: Send FAILED: 1780898356757935 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-08 Mon 05:59:16:758.125 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-08 Mon 05:59:16:764.684 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-08 Mon 05:59:16:764.811 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:59:16:764.822 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-08 Mon 05:59:16:764.841 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: SANDESH: Send FAILED: 1780898356764781 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-08 Mon 05:59:16:764.847 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-08 Mon 05:59:16:765.948 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-08 Mon 05:59:16:765.977 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-08 Mon 05:59:16:765.989 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-08 Mon 05:59:16:766.000 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-08 Mon 05:59:16:766.059 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-08 Mon 05:59:16:766.068 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-08 Mon 05:59:16:766.080 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-08 Mon 05:59:16:766.086 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-08 Mon 05:59:16:766.097 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-08 Mon 05:59:16:766.112 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-08 Mon 05:59:16:766.123 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-08 Mon 05:59:16:766.130 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-08 Mon 05:59:16:766.140 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-08 Mon 05:59:16:766.147 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-08 Mon 05:59:16:766.158 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-08 Mon 05:59:16:766.165 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-08 Mon 05:59:16:766.177 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-08 Mon 05:59:16:766.187 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-08 Mon 05:59:16:766.202 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-08 Mon 05:59:16:766.212 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-08 Mon 05:59:16:766.226 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-08 Mon 05:59:16:766.236 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-08 Mon 05:59:16:766.251 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-08 Mon 05:59:16:766.259 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-08 Mon 05:59:16:766.280 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-08 Mon 05:59:16:767.115 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:59:16:767.139 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-08 Mon 05:59:16:767.156 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: SANDESH: Send FAILED: 1780898356766305 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-08 Mon 05:59:16:767.163 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-08 Mon 05:59:16:767.295 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-08 Mon 05:59:16:767.677 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-08 Mon 05:59:16:767.684 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-08 Mon 05:59:16:767.697 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: SANDESH: Send FAILED: 1780898356767315 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-08 Mon 05:59:16:767.709 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-08 Mon 05:59:16:767.720 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-08 Mon 05:59:16:767.732 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-08 Mon 05:59:16:767.741 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-08 Mon 05:59:16:767.750 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-08 Mon 05:59:16:767.759 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-08 Mon 05:59:16:767.772 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-08 Mon 05:59:16:767.781 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-08 Mon 05:59:16:767.797 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-08 Mon 05:59:16:768.006 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: OnMessage control in state: ClientInit 2026-06-08 Mon 05:59:16:768.027 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-06-08 Mon 05:59:16:768.033 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-06-08 Mon 05:59:16:768.047 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Received Ctrl Message with size 0 2026-06-08 Mon 05:59:16:768.123 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5902-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780898356649485 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 21, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2026-06-08 Mon 05:59:16:768.142 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-06-08 Mon 05:59:16:768.159 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5902-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2026-06-08 Mon 05:59:16:768.168 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-06-08 Mon 05:59:16:768.174 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-06-08 Mon 05:59:16:768.180 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-06-08 Mon 05:59:16:768.201 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: Established 2026-06-08 Mon 06:00:16:650.146 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-06-08 Mon 06:01:16:651.588 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211139647040, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-06-08 Mon 06:02:16:654.804 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210272720448, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-06-08 Mon 06:03:16:655.485 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210281117248, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-06-08 Mon 06:04:16:656.519 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211131250240, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-06-08 Mon 06:05:16:657.789 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211135448640, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2026-06-08 Mon 06:06:16:658.726 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210276918848, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2026-06-08 Mon 06:07:16:659.305 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210260125248, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2026-06-08 Mon 06:08:16:659.954 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210272720448, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2026-06-08 Mon 06:09:16:661.280 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210264323648, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2026-06-08 Mon 06:10:16:663.553 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210268522048, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2026-06-08 Mon 06:11:16:664.121 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211122853440, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2026-06-08 Mon 06:12:16:664.737 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211139647040, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2026-06-08 Mon 06:13:16:665.371 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211131250240, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2026-06-08 Mon 06:14:16:666.036 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210281117248, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2026-06-08 Mon 06:14:57:754.506 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.50:39200 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-06-08 Mon 06:15:03:072.755 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.50:37616 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-06-08 Mon 06:15:16:666.607 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211139647040, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2026-06-08 Mon 06:16:16:667.137 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211152242240, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2026-06-08 Mon 06:16:30:777.793 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.38:57800 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-06-08 Mon 06:16:53:442.898 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.38 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 812 2026-06-08 Mon 06:16:53:443.599 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID a4b15344-6301-11f1-8517-00000a000026 src/contrail-analytics/contrail-query-engine/query.cc 1165 2026-06-08 Mon 06:16:53:443.650 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QEQueryLog: a4b15344-6301-11f1-8517-00000a000026 :0:json query is: { end_time: 1780899413376722 , enqueue_time: 1780899413441894 , query_metadata: {"enqueue_time": 1780899413441802} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1780899394997104 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "2ee79f07-933d-4257-a671-8c4bcb2766a9", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-06-08 Mon 06:16:53:443.854 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 936 2026-06-08 Mon 06:16:53:443.871 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 986 2026-06-08 Mon 06:16:53:443.935 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for a4b15344-6301-11f1-8517-00000a000026 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 994 2026-06-08 Mon 06:16:53:444.046 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211131250240, Pid 1]: [SYS_INFO]: QEQueryLog: a4b15344-6301-11f1-8517-00000a000026 :0:json query is: { end_time: 1780899413376722 , enqueue_time: 1780899413441894 , query_metadata: {"enqueue_time": 1780899413441802} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1780899394997104 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "2ee79f07-933d-4257-a671-8c4bcb2766a9", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-06-08 Mon 06:16:53:488.293 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211148043840, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:53:488.340 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211148043840, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:53:488.350 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211148043840, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:53:488.368 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211148043840, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:53:488.374 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211148043840, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:53:488.380 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211148043840, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:53:488.386 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211148043840, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:53:488.391 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211148043840, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:53:488.397 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211148043840, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:53:488.403 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211148043840, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:53:489.181 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:53:489.200 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:53:489.207 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:53:489.213 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:53:489.219 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:53:489.225 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:53:489.231 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 UNION between tables of sizes 0 and 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:53:489.248 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 Resulting size of set 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:53:489.263 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 UNION between tables of sizes 3 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:53:489.272 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 Resulting size of set 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:53:489.315 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QEQueryLog: a4b15344-6301-11f1-8517-00000a000026 :0:json query is: { end_time: 1780899413376722 , enqueue_time: 1780899413441894 , query_metadata: {"enqueue_time": 1780899413441802} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1780899394997104 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "2ee79f07-933d-4257-a671-8c4bcb2766a9", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-06-08 Mon 06:16:53:489.465 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QEQueryLog: a4b15344-6301-11f1-8517-00000a000026 ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id 2ee79f07-933d-4257-a671-8c4bcb2766a9 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-06-08 Mon 06:16:53:489.565 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QEQueryLog: a4b15344-6301-11f1-8517-00000a000026 ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id 2ee79f07-933d-4257-a671-8c4bcb2766a9 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-06-08 Mon 06:16:53:489.660 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QEQueryLog: a4b15344-6301-11f1-8517-00000a000026 ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id 2ee79f07-933d-4257-a671-8c4bcb2766a9 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-06-08 Mon 06:16:53:490.180 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:53:490.223 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:53:490.232 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:53:490.242 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:53:490.249 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:53:490.274 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:53:490.281 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:53:490.288 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:53:490.295 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:53:490.303 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a4b15344-6301-11f1-8517-00000a000026 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:53:490.538 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QEQueryLog: a4b15344-6301-11f1-8517-00000a000026 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 2ee79f07-933d-4257-a671-8c4bcb2766a9 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-06-08 Mon 06:16:53:490.659 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QEQueryLog: a4b15344-6301-11f1-8517-00000a000026 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 2ee79f07-933d-4257-a671-8c4bcb2766a9 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-06-08 Mon 06:16:53:490.897 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 542 2026-06-08 Mon 06:16:53:490.967 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 551 2026-06-08 Mon 06:16:53:491.004 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":5} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 586 2026-06-08 Mon 06:16:53:491.448 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: Finished: QID a4b15344-6301-11f1-8517-00000a000026 Table ObjectVMTable Time(ms) 47 RedisTime(ms) 0 MergeTime(ms) 0 Rows 5 EnQ-delay1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 661 2026-06-08 Mon 06:16:53:491.493 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211139647040, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-5902-2. table = ObjectVMTable query_stats= [ qid = a4b15344-6301-11f1-8517-00000a000026 where = [[{"name": "ObjectId", "value": "2ee79f07-933d-4257-a671-8c4bcb2766a9", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 45,0, 44,0, 44,0, chunk_select_time = 0,0, 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 47 rows = 5 enq_delay = 1 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 657 2026-06-08 Mon 06:16:53:491.522 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 699 2026-06-08 Mon 06:16:56:528.168 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.38 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 812 2026-06-08 Mon 06:16:56:529.055 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID a687f85f-6301-11f1-9217-00000a000026 src/contrail-analytics/contrail-query-engine/query.cc 1165 2026-06-08 Mon 06:16:56:529.107 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QEQueryLog: a687f85f-6301-11f1-9217-00000a000026 :0:json query is: { end_time: 1780899416512060 , enqueue_time: 1780899416527067 , query_metadata: {"enqueue_time": 1780899416526967} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1780899394997104 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-60644681:ctest-vn22-14615055:ctest-vn22-14615055", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-06-08 Mon 06:16:56:529.239 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 936 2026-06-08 Mon 06:16:56:529.255 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 986 2026-06-08 Mon 06:16:56:529.303 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for a687f85f-6301-11f1-9217-00000a000026 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 994 2026-06-08 Mon 06:16:56:529.407 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211139647040, Pid 1]: [SYS_INFO]: QEQueryLog: a687f85f-6301-11f1-9217-00000a000026 :0:json query is: { end_time: 1780899416512060 , enqueue_time: 1780899416527067 , query_metadata: {"enqueue_time": 1780899416526967} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1780899394997104 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-60644681:ctest-vn22-14615055:ctest-vn22-14615055", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-06-08 Mon 06:16:56:579.888 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210276918848, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:56:579.950 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210276918848, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:56:579.959 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210276918848, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:56:579.966 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210276918848, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:56:579.972 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210276918848, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:56:579.979 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210276918848, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:56:579.985 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210276918848, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:56:579.992 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210276918848, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:56:579.998 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210276918848, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:56:580.005 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210276918848, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:56:581.060 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:56:581.087 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:56:581.095 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:56:581.103 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:56:581.110 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:56:581.117 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:56:581.139 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:56:581.147 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:56:581.153 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:56:581.160 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:56:581.444 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211143845440, Pid 1]: [SYS_INFO]: QEQueryLog: a687f85f-6301-11f1-9217-00000a000026 ObjectRoutingInstance :1:Column ObjectTypeName1 Table ObjectRoutingInstance object_id default-domain:ctest-AnalyticsBasicTestSanity-60644681:ctest-vn22-14615055:ctest-vn22-14615055 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-06-08 Mon 06:16:56:581.658 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210264323648, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:56:581.670 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210264323648, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:56:581.677 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210264323648, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:56:581.731 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210264323648, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:56:581.740 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210264323648, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:56:581.746 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210264323648, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:56:581.752 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210264323648, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:56:581.758 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210264323648, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:56:581.764 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210264323648, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-08 Mon 06:16:56:581.770 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210264323648, Pid 1]: [SYS_INFO]: QELog: a687f85f-6301-11f1-9217-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-08 Mon 06:16:56:581.889 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210281117248, Pid 1]: [SYS_INFO]: QEQueryLog: a687f85f-6301-11f1-9217-00000a000026 :0:json query is: { end_time: 1780899416512060 , enqueue_time: 1780899416527067 , query_metadata: {"enqueue_time": 1780899416526967} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1780899394997104 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-60644681:ctest-vn22-14615055:ctest-vn22-14615055", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-06-08 Mon 06:16:56:582.118 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210264323648, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 542 2026-06-08 Mon 06:16:56:582.149 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210264323648, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 551 2026-06-08 Mon 06:16:56:582.220 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210264323648, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":1} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 586 2026-06-08 Mon 06:16:56:583.810 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211135448640, Pid 1]: [SYS_INFO]: QELog: Finished: QID a687f85f-6301-11f1-9217-00000a000026 Table ObjectRoutingInstance Time(ms) 54 RedisTime(ms) 0 MergeTime(ms) 0 Rows 1 EnQ-delay1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 661 2026-06-08 Mon 06:16:56:583.888 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211135448640, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 699 2026-06-08 Mon 06:16:56:583.951 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211135448640, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-5902-2. table = ObjectRoutingInstance query_stats= [ qid = a687f85f-6301-11f1-9217-00000a000026 where = [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-60644681:ctest-vn22-14615055:ctest-vn22-14615055", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 51,0, 51,0, 46,0, chunk_select_time = 0,0, 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 54 rows = 1 enq_delay = 1 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 657 2026-06-08 Mon 06:17:16:667.769 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211148043840, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2026-06-08 Mon 06:18:16:668.502 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210281117248, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2026-06-08 Mon 06:19:16:669.266 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211135448640, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2026-06-08 Mon 06:20:16:669.920 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211139647040, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2026-06-08 Mon 06:21:16:670.834 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210281117248, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2026-06-08 Mon 06:22:16:671.456 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210281117248, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2026-06-08 Mon 06:23:16:672.747 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211131250240, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2026-06-08 Mon 06:24:16:673.392 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210276918848, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2026-06-08 Mon 06:25:16:674.055 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210281117248, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2026-06-08 Mon 06:26:16:674.696 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210268522048, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2026-06-08 Mon 06:27:16:675.833 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140210264323648, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2026-06-08 Mon 06:27:48:197.211 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.38:52474 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-06-08 Mon 06:28:16:676.543 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211156440640, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0 2026-06-08 Mon 06:29:16:677.746 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211152242240, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 30 without seqno , total = 0 2026-06-08 Mon 06:30:16:678.485 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211139647040, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 31 without seqno , total = 0 2026-06-08 Mon 06:30:30:206.782 UTC cn-jenkins-deploy-platform-ansible-os-5902-2 [Thread 140211167778944, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.50:53106 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256