2026-06-04 Thu 05:58:36:001.550 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: http-server-port 8091 2026-06-04 Thu 05:58:36:001.641 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Max-tasks 16 2026-06-04 Thu 05:58:36:001.650 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Max-slice 100 2026-06-04 Thu 05:58:36:001.657 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Collectors 10.0.0.38:8086 2026-06-04 Thu 05:58:36:001.663 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Collectors 10.0.0.50:8086 2026-06-04 Thu 05:58:36:001.669 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Collectors 10.0.0.33:8086 2026-06-04 Thu 05:58:36:001.700 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: SANDESH: ROLE : Generator 2026-06-04 Thu 05:58:36:001.707 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-06-04 Thu 05:58:36:001.713 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5888-2. 2026-06-04 Thu 05:58:36:001.719 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: SANDESH: NODE TYPE : Database 2026-06-04 Thu 05:58:36:001.725 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: SANDESH: INSTANCE ID : 0 2026-06-04 Thu 05:58:36:001.731 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-06-04 Thu 05:58:36:001.738 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-06-04 Thu 05:58:36:005.696 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: HTTP Introspect Init 2026-06-04 Thu 05:58:36:005.864 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2026-06-04 Thu 05:58:36:005.990 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Sandesh Http Server Port: 8091 2026-06-04 Thu 05:58:36:006.068 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-06-04 Thu 05:58:36:006.080 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-06-04 Thu 05:58:36:006.521 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Idle 2026-06-04 Thu 05:58:36:006.657 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: SANDESH: No Client: 1780552716006600 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780552716006533 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 05:58:36:006.779 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-06-04 Thu 05:58:36:006.792 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-06-04 Thu 05:58:36:006.802 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Cassandra Servers: 10.0.0.38:9042 10.0.0.50:9042 10.0.0.33:9042 2026-06-04 Thu 05:58:36:010.958 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: Processing scm::EvStart in state Idle 2026-06-04 Thu 05:58:36:011.028 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: Disconnect 2026-06-04 Thu 05:58:36:011.092 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-06-04 Thu 05:58:36:011.267 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-06-04 Thu 05:58:36:011.336 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 05:58:36:011.358 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: SANDESH: Send FAILED: 1780552716011020 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-06-04 Thu 05:58:36:011.365 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 05:58:36:011.372 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 05:58:36:011.455 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: SANDESH: Send FAILED: 1780552716011082 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780552716006533 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 05:58:36:011.476 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 05:58:36:011.484 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 05:58:36:011.530 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: SANDESH: Send FAILED: 1780552716011121 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780552716006533 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 05:58:36:011.540 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 05:58:36:011.547 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 05:58:36:011.558 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: SANDESH: Send FAILED: 1780552716011137 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-06-04 Thu 05:58:36:011.563 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 05:58:36:011.569 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 05:58:36:011.607 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: SANDESH: Send FAILED: 1780552716011322 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780552716006533 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 05:58:36:011.616 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 05:58:36:012.625 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-06-04 Thu 05:58:36:012.648 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-06-04 Thu 05:58:36:012.698 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-06-04 Thu 05:58:36:012.732 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446839862848, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 05:58:36:012.760 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446839862848, Pid 1]: SANDESH: Send FAILED: 1780552716012678 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-06-04 Thu 05:58:36:012.769 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446839862848, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 05:58:41:016.032 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-06-04 Thu 05:58:41:016.135 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-06-04 Thu 05:58:41:016.196 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-06-04 Thu 05:58:46:018.588 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-06-04 Thu 05:58:46:018.632 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-06-04 Thu 05:58:46:018.649 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-06-04 Thu 05:58:51:020.979 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-06-04 Thu 05:58:51:021.021 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-06-04 Thu 05:58:51:021.039 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-06-04 Thu 05:58:56:210.989 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2026-06-04 Thu 05:58:56:211.049 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-06-04 Thu 05:58:56:211.073 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-06-04 Thu 05:59:01:260.306 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2026-06-04 Thu 05:59:01:260.362 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-06-04 Thu 05:59:01:260.385 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-06-04 Thu 05:59:06:320.406 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2026-06-04 Thu 05:59:06:320.467 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-06-04 Thu 05:59:06:320.486 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-06-04 Thu 05:59:11:357.082 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2026-06-04 Thu 05:59:11:357.142 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-06-04 Thu 05:59:11:357.171 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 7 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-06-04 Thu 05:59:16:385.180 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-06-04 Thu 05:59:16:385.240 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 8 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-06-04 Thu 05:59:21:432.841 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-06-04 Thu 05:59:21:432.899 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 9 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-06-04 Thu 05:59:26:478.445 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-06-04 Thu 05:59:31:484.246 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: RAC_Connect status: 0 0x1ebf840 2026-06-04 Thu 05:59:31:484.279 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Connected to REDIS... 2026-06-04 Thu 05:59:31:484.293 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 05:59:31:484.303 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: RAC_Connect status: 0 0x1ebd180 2026-06-04 Thu 05:59:31:484.308 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Connected to REDIS... 2026-06-04 Thu 05:59:31:484.314 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 05:59:31:484.321 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: RAC_Connect status: 0 0x1ebe200 2026-06-04 Thu 05:59:31:484.326 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Connected to REDIS... 2026-06-04 Thu 05:59:31:484.332 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 05:59:31:484.339 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: RAC_Connect status: 0 0x1ec0360 2026-06-04 Thu 05:59:31:484.343 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Connected to REDIS... 2026-06-04 Thu 05:59:31:484.349 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 05:59:31:484.356 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: RAC_Connect status: 0 0x1eb81c0 2026-06-04 Thu 05:59:31:484.360 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Connected to REDIS... 2026-06-04 Thu 05:59:31:484.372 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 05:59:31:484.379 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: RAC_Connect status: 0 0x1ebed20 2026-06-04 Thu 05:59:31:484.384 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Connected to REDIS... 2026-06-04 Thu 05:59:31:484.390 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 05:59:31:484.396 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: RAC_Connect status: 0 0x1ebd8e0 2026-06-04 Thu 05:59:31:484.400 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Connected to REDIS... 2026-06-04 Thu 05:59:31:484.406 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 05:59:31:484.413 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: RAC_Connect status: 0 0x1ec0f20 2026-06-04 Thu 05:59:31:484.417 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Connected to REDIS... 2026-06-04 Thu 05:59:31:484.423 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 05:59:31:484.430 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: RAC_Connect status: 0 0x1ec1a00 2026-06-04 Thu 05:59:31:484.434 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Connected to REDIS... 2026-06-04 Thu 05:59:31:484.440 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 05:59:31:484.447 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: RAC_Connect status: 0 0x1ec24e0 2026-06-04 Thu 05:59:31:484.451 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Connected to REDIS... 2026-06-04 Thu 05:59:31:484.457 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 05:59:31:484.463 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: RAC_Connect status: 0 0x1ec2fc0 2026-06-04 Thu 05:59:31:484.467 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Connected to REDIS... 2026-06-04 Thu 05:59:31:484.473 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 05:59:31:484.480 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: RAC_Connect status: 0 0x1ec3aa0 2026-06-04 Thu 05:59:31:484.484 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Connected to REDIS... 2026-06-04 Thu 05:59:31:484.490 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 05:59:31:484.496 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: RAC_Connect status: 0 0x1ec4580 2026-06-04 Thu 05:59:31:484.502 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Connected to REDIS... 2026-06-04 Thu 05:59:31:484.508 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 05:59:31:484.515 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: RAC_Connect status: 0 0x1ec5060 2026-06-04 Thu 05:59:31:484.519 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Connected to REDIS... 2026-06-04 Thu 05:59:31:484.525 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 05:59:31:484.531 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: RAC_Connect status: 0 0x1ec5b40 2026-06-04 Thu 05:59:31:484.535 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: Connected to REDIS... 2026-06-04 Thu 05:59:31:484.541 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 05:59:31:484.560 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-06-04 Thu 05:59:31:484.565 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: -(-1):EvTcpConnectFail 2026-06-04 Thu 05:59:31:484.765 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 05:59:31:484.792 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: SANDESH: Send FAILED: 1780552771484129 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-06-04 Thu 05:59:31:484.800 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 05:59:31:484.807 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-06-04 Thu 05:59:31:484.813 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: Connect : EvTcpConnectFail 2026-06-04 Thu 05:59:31:484.909 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2026-06-04 Thu 05:59:31:484.942 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: Idle 2026-06-04 Thu 05:59:31:484.991 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 05:59:31:485.060 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: SANDESH: Send FAILED: 1780552771484870 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780552716006533 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 44 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 45, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 05:59:31:485.076 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 05:59:31:485.084 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 05:59:31:485.099 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: SANDESH: Send FAILED: 1780552771484894 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-06-04 Thu 05:59:31:486.324 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 05:59:31:486.339 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-04 Thu 05:59:31:486.384 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 05:59:31:486.478 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: SANDESH: Send FAILED: 1780552771484981 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780552716006533 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 44 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 46, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 05:59:31:486.492 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 05:59:31:486.572 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 05:59:31:486.606 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: SANDESH: Send FAILED: 1780552771486518 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1780552771484941 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1780552771486383 ] max_sm_queue_count = 5 ] 2026-06-04 Thu 05:59:31:486.612 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 05:59:31:486.619 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 05:59:31:486.702 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: SANDESH: Send FAILED: 1780552771486560 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780552716006533 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 44 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 46, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 10, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 05:59:31:547.551 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446810474048, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 05:59:31:487.460 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 05:59:31:547.799 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 05:59:31:547.837 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 05:59:31:547.849 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 05:59:31:547.867 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 05:59:31:547.878 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 05:59:31:547.894 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 05:59:31:547.905 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 05:59:31:547.923 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 05:59:31:547.933 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 05:59:31:547.951 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 05:59:31:547.961 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 05:59:31:547.978 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 05:59:31:547.989 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 05:59:31:548.004 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 05:59:31:548.020 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 05:59:31:548.036 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 05:59:31:548.046 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 05:59:31:548.064 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 05:59:31:548.075 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 05:59:31:548.092 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 05:59:31:548.118 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 05:59:31:548.134 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 05:59:31:548.158 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 05:59:31:548.176 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 05:59:31:548.186 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 05:59:31:548.202 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 05:59:31:548.212 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 05:59:31:548.229 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 05:59:31:548.239 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 05:59:31:548.331 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 05:59:31:548.369 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140445829031488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 05:59:31:548.416 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140445829031488, Pid 1]: SANDESH: Send FAILED: 1780552771548289 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 05:59:31:548.426 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140445829031488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 05:59:31:548.438 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140445829031488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 05:59:31:548.460 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140445829031488, Pid 1]: SANDESH: Send FAILED: 1780552771548363 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 05:59:31:548.468 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140445829031488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 05:59:31:548.370 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 05:59:31:549.141 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 05:59:31:549.172 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 05:59:31:549.190 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 05:59:31:549.220 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 05:59:31:549.266 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 05:59:31:549.282 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 05:59:31:549.305 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 05:59:31:549.322 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 05:59:31:549.350 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 05:59:31:549.366 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 05:59:31:549.386 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 05:59:31:549.401 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 05:59:31:549.415 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 05:59:31:549.573 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 05:59:31:549.610 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: SANDESH: Send FAILED: 1780552771549246 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 05:59:31:549.620 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 05:59:31:549.631 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 05:59:31:549.651 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: SANDESH: Send FAILED: 1780552771549343 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 05:59:31:549.660 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446929860160, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 05:59:32:485.732 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446831466048, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-06-04 Thu 05:59:32:486.136 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446831466048, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-06-04 Thu 05:59:32:487.070 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446831466048, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 05:59:32:487.112 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446831466048, Pid 1]: SANDESH: Send FAILED: 1780552772485802 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 05:59:32:487.124 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446831466048, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 05:59:32:487.132 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446831466048, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 05:59:32:487.261 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446831466048, Pid 1]: SANDESH: Send FAILED: 1780552772487047 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780552716006533 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 89 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 05:59:32:487.281 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446831466048, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 05:59:32:488.386 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:47332::10.0.0.50:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 281 2026-06-04 Thu 05:59:32:488.406 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: 10.0.0.50:47332::10.0.0.50:8086(-1) OnSessionEvent TCP Connected 2026-06-04 Thu 05:59:32:488.414 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: 10.0.0.50:47332::10.0.0.50:8086(-1):EvTcpConnected 2026-06-04 Thu 05:59:32:488.486 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446818870848, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-06-04 Thu 05:59:32:488.499 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446818870848, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-06-04 Thu 05:59:32:488.529 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446818870848, Pid 1]: ClientInit 2026-06-04 Thu 05:59:32:488.545 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446818870848, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5888-2.:contrail-query-engine:0:Database count 1 2026-06-04 Thu 05:59:32:488.979 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446818870848, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 05:59:32:489.806 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446818870848, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-04 Thu 05:59:32:489.862 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446818870848, Pid 1]: SANDESH: Send FAILED: 1780552772488524 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 05:59:32:489.870 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446818870848, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-04 Thu 05:59:32:489.880 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446818870848, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 05:59:32:489.886 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446818870848, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-06-04 Thu 05:59:32:489.907 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446818870848, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 05:59:32:489.913 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446818870848, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-04 Thu 05:59:32:489.990 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446818870848, Pid 1]: SANDESH: Send FAILED: 1780552772488602 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780552716006533 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 89 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 05:59:32:490.005 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446818870848, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-04 Thu 05:59:32:499.435 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446839862848, Pid 1]: OnMessage control in state: ClientInit 2026-06-04 Thu 05:59:32:499.490 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446839862848, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-06-04 Thu 05:59:32:499.506 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446839862848, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-06-04 Thu 05:59:32:499.523 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446839862848, Pid 1]: Received Ctrl Message with size 0 2026-06-04 Thu 05:59:32:499.633 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446839862848, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780552716006533 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1780552771484941 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1780552771486383 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 89 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 5 proxy seq 10 2026-06-04 Thu 05:59:32:499.666 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446839862848, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-06-04 Thu 05:59:32:499.686 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446839862848, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2026-06-04 Thu 05:59:32:499.697 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446839862848, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-06-04 Thu 05:59:32:499.704 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446839862848, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-06-04 Thu 05:59:32:499.711 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446839862848, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-06-04 Thu 05:59:32:499.744 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446839862848, Pid 1]: Established 2026-06-04 Thu 05:59:36:948.202 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446785291840, Pid 1]: OnMessage control in state: Established 2026-06-04 Thu 05:59:36:948.295 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446785291840, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2026-06-04 Thu 05:59:36:948.305 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446785291840, Pid 1]: Established : EvSandeshMessageRecv 2026-06-04 Thu 05:59:36:948.331 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446785291840, Pid 1]: Received Ctrl Message with size 3 2026-06-04 Thu 05:59:36:948.344 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446785291840, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 11, total = 0 2026-06-04 Thu 05:59:36:948.353 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446785291840, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 13, total = 0 2026-06-04 Thu 05:59:36:948.365 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446785291840, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2026-06-04 Thu 05:59:36:948.371 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446785291840, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-06-04 Thu 06:00:01:487.105 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446914573888, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-06-04 Thu 06:01:01:487.868 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446785291840, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-06-04 Thu 06:02:01:488.690 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446785291840, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-06-04 Thu 06:03:01:489.561 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446781093440, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-06-04 Thu 06:03:08:297.909 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.50:42690 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2026-06-04 Thu 06:03:19:793.793 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446941200896, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.50:38494 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2026-06-04 Thu 06:04:01:490.268 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140445833229888, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-06-04 Thu 06:05:01:492.612 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140446781093440, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2026-06-04 Thu 06:05:20:957.203 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140445829031488, Pid 1]: Processing scm::EvStop in state Established 2026-06-04 Thu 06:05:20:957.302 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140445829031488, Pid 1]: Idle 2026-06-04 Thu 06:05:20:957.419 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140445829031488, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-04 Thu 06:05:20:957.431 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140445829031488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:05:20:957.554 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140445829031488, Pid 1]: SANDESH: Send FAILED: 1780553120957390 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780552716006533 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 306019 calls = 33 average_bytes = 9273 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 89 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 24 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 94, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 34, ] ] ] 2026-06-04 Thu 06:05:20:957.578 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140445829031488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:05:36:086.874 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: http-server-port 8091 2026-06-04 Thu 06:05:36:086.994 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Max-tasks 16 2026-06-04 Thu 06:05:36:087.001 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Max-slice 100 2026-06-04 Thu 06:05:36:087.006 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Collectors 10.0.0.38:8086 2026-06-04 Thu 06:05:36:087.011 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Collectors 10.0.0.50:8086 2026-06-04 Thu 06:05:36:087.015 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Collectors 10.0.0.33:8086 2026-06-04 Thu 06:05:36:087.188 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: SANDESH: ROLE : Generator 2026-06-04 Thu 06:05:36:087.202 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-06-04 Thu 06:05:36:087.209 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5888-2. 2026-06-04 Thu 06:05:36:087.227 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: SANDESH: NODE TYPE : Database 2026-06-04 Thu 06:05:36:088.510 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: SANDESH: INSTANCE ID : 0 2026-06-04 Thu 06:05:36:088.524 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-06-04 Thu 06:05:36:088.530 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-06-04 Thu 06:05:36:091.789 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: HTTP Introspect Init 2026-06-04 Thu 06:05:36:091.896 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2026-06-04 Thu 06:05:36:091.980 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Sandesh Http Server Port: 8091 2026-06-04 Thu 06:05:36:092.019 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-06-04 Thu 06:05:36:092.027 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-06-04 Thu 06:05:36:092.443 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Idle 2026-06-04 Thu 06:05:36:092.575 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: SANDESH: No Client: 1780553136092520 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 06:05:36:092.683 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-06-04 Thu 06:05:36:092.695 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-06-04 Thu 06:05:36:092.705 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Cassandra Servers: 10.0.0.38:9042 10.0.0.50:9042 10.0.0.33:9042 2026-06-04 Thu 06:05:36:092.805 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Processing scm::EvStart in state Idle 2026-06-04 Thu 06:05:36:092.872 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Disconnect 2026-06-04 Thu 06:05:36:092.939 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-06-04 Thu 06:05:36:093.183 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-06-04 Thu 06:05:36:093.243 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:05:36:093.260 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: SANDESH: Send FAILED: 1780553136092865 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-06-04 Thu 06:05:36:093.268 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:05:36:093.276 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:05:36:093.341 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: SANDESH: Send FAILED: 1780553136092925 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 06:05:36:093.351 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:05:36:093.359 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:05:36:093.415 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: SANDESH: Send FAILED: 1780553136092980 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 06:05:36:093.428 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:05:36:093.435 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:05:36:093.447 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: SANDESH: Send FAILED: 1780553136093003 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-06-04 Thu 06:05:36:093.453 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:05:36:093.459 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:05:36:093.498 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: SANDESH: Send FAILED: 1780553136093231 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 06:05:36:095.116 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:05:36:168.982 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-06-04 Thu 06:05:36:174.209 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-06-04 Thu 06:05:36:174.240 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: -(-1):EvTcpConnectFail 2026-06-04 Thu 06:05:36:174.261 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd4770 2026-06-04 Thu 06:05:36:174.293 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5290 2026-06-04 Thu 06:05:36:174.306 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5e50 2026-06-04 Thu 06:05:36:174.317 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd6930 2026-06-04 Thu 06:05:36:174.329 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7410 2026-06-04 Thu 06:05:36:174.340 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbcd180 2026-06-04 Thu 06:05:36:174.344 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:05:36:174.366 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:05:36:174.375 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd2180 2026-06-04 Thu 06:05:36:174.379 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:05:36:174.386 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:05:36:174.392 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd3130 2026-06-04 Thu 06:05:36:174.397 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:05:36:174.403 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:05:36:174.411 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd2830 2026-06-04 Thu 06:05:36:174.415 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:05:36:174.421 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:05:36:174.428 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd3c50 2026-06-04 Thu 06:05:36:174.433 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:05:36:174.439 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:05:36:174.447 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7ef0 2026-06-04 Thu 06:05:36:174.459 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd89d0 2026-06-04 Thu 06:05:36:174.471 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd94b0 2026-06-04 Thu 06:05:36:174.484 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd9f90 2026-06-04 Thu 06:05:36:174.496 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbdaa70 2026-06-04 Thu 06:05:36:174.648 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:05:36:175.235 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:05:36:175.286 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:05:36:175.298 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:05:36:175.305 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:05:36:175.321 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:05:36:175.328 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:05:36:175.337 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:05:36:175.344 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:05:36:175.353 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:05:36:175.360 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:05:36:175.404 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:05:36:175.425 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:05:36:175.436 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:05:36:175.447 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:05:36:175.466 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:05:36:175.536 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: SANDESH: Send FAILED: 1780553136174103 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-06-04 Thu 06:05:36:175.553 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:05:36:175.562 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-06-04 Thu 06:05:36:175.568 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Connect : EvTcpConnectFail 2026-06-04 Thu 06:05:36:175.666 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2026-06-04 Thu 06:05:36:175.699 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Idle 2026-06-04 Thu 06:05:36:175.758 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:05:36:175.776 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: SANDESH: Send FAILED: 1780553136175394 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 06:05:36:175.782 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:05:36:175.788 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:05:36:175.800 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: SANDESH: Send FAILED: 1780553136175421 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 06:05:36:175.809 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:05:36:175.815 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:05:36:175.880 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: SANDESH: Send FAILED: 1780553136175628 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 22, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 06:05:36:175.893 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:05:36:175.901 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:05:36:175.913 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: SANDESH: Send FAILED: 1780553136175645 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 06:05:36:175.921 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:05:36:175.928 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-04 Thu 06:05:36:175.966 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:05:36:176.030 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: SANDESH: Send FAILED: 1780553136175748 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 23, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 06:05:36:176.041 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568121386560, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:05:37:175.829 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567027639872, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-06-04 Thu 06:05:37:176.423 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567027639872, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-06-04 Thu 06:05:37:176.516 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567027639872, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:05:37:176.543 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567027639872, Pid 1]: SANDESH: Send FAILED: 1780553137175893 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 06:05:37:176.551 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567027639872, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:05:37:176.559 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567027639872, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:05:37:176.648 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567027639872, Pid 1]: SANDESH: Send FAILED: 1780553137176502 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 23, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 11, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 06:05:37:176.676 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567027639872, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:05:37:178.389 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:40720::10.0.0.33:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 281 2026-06-04 Thu 06:05:37:178.414 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:40720::10.0.0.33:8086(-1) OnSessionEvent TCP Connected 2026-06-04 Thu 06:05:37:178.422 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:40720::10.0.0.33:8086(-1):EvTcpConnected 2026-06-04 Thu 06:05:37:178.484 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-06-04 Thu 06:05:37:178.495 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-06-04 Thu 06:05:37:178.531 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: ClientInit 2026-06-04 Thu 06:05:37:178.860 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5888-2.:contrail-query-engine:0:Database count 1 2026-06-04 Thu 06:05:37:178.939 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:05:37:178.946 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-04 Thu 06:05:37:178.967 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: SANDESH: Send FAILED: 1780553137178525 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 06:05:37:178.974 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-04 Thu 06:05:37:178.981 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:05:37:178.986 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-06-04 Thu 06:05:37:179.006 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:05:37:179.012 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-04 Thu 06:05:37:179.088 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: SANDESH: Send FAILED: 1780553137178928 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 24, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 13, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 06:05:37:179.110 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-04 Thu 06:05:39:178.626 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: 10.0.0.33:8086 EvConnectTimerExpired in state ClientInit 2026-06-04 Thu 06:05:39:178.683 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvConnectTimerExpired in state ClientInit 2026-06-04 Thu 06:05:39:178.691 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: ClientInit : EvConnectTimerExpired 2026-06-04 Thu 06:05:39:179.328 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Return to idle with 1000 2026-06-04 Thu 06:05:39:179.341 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Idle 2026-06-04 Thu 06:05:39:179.399 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:05:39:179.492 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: SANDESH: Send FAILED: 1780553139178752 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 24, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 15, _iter59->first = sent _iter59->second = 1, ] ] ] 2026-06-04 Thu 06:05:39:179.512 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:05:39:179.522 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:05:39:179.538 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: SANDESH: Send FAILED: 1780553139178783 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Down description = ClientInit : EvConnectTimerExpired ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 06:05:39:179.547 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:05:39:179.554 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-04 Thu 06:05:39:179.585 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:05:39:179.651 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: SANDESH: Send FAILED: 1780553139179387 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 24, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 15, _iter59->first = sent _iter59->second = 1, ] ] ] 2026-06-04 Thu 06:05:39:179.662 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:05:40:179.440 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-06-04 Thu 06:05:40:179.672 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-06-04 Thu 06:05:40:179.760 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:05:40:179.784 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SANDESH: Send FAILED: 1780553140179506 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 06:05:40:179.794 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:05:40:179.801 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:05:40:179.892 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SANDESH: Send FAILED: 1780553140179745 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 24, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 1, ] ] ] 2026-06-04 Thu 06:05:40:179.916 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:05:40:182.286 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:57004::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 281 2026-06-04 Thu 06:05:40:182.307 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:57004::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2026-06-04 Thu 06:05:40:182.314 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:57004::10.0.0.38:8086(-1):EvTcpConnected 2026-06-04 Thu 06:05:40:182.722 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-06-04 Thu 06:05:40:182.734 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-06-04 Thu 06:05:40:182.769 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: ClientInit 2026-06-04 Thu 06:05:40:182.779 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5888-2.:contrail-query-engine:0:Database count 2 2026-06-04 Thu 06:05:40:182.851 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:05:40:182.858 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-04 Thu 06:05:40:182.876 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: SANDESH: Send FAILED: 1780553140182763 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 06:05:40:182.882 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-04 Thu 06:05:40:182.889 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:05:40:182.901 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-06-04 Thu 06:05:40:182.919 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:05:40:182.924 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-04 Thu 06:05:40:183.005 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: SANDESH: Send FAILED: 1780553140182840 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 25, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 1, ] ] ] 2026-06-04 Thu 06:05:40:183.021 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-04 Thu 06:05:40:188.415 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: OnMessage control in state: ClientInit 2026-06-04 Thu 06:05:40:188.466 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-06-04 Thu 06:05:40:188.474 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-06-04 Thu 06:05:40:188.491 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Received Ctrl Message with size 0 2026-06-04 Thu 06:05:40:188.591 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 25, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 1, ] ] proxy seq 12 2026-06-04 Thu 06:05:40:188.619 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-06-04 Thu 06:05:40:188.644 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2026-06-04 Thu 06:05:40:188.654 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-06-04 Thu 06:05:40:188.660 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-06-04 Thu 06:05:40:188.667 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-06-04 Thu 06:05:40:188.696 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Established 2026-06-04 Thu 06:05:41:175.380 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd4770 2026-06-04 Thu 06:05:41:175.558 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5290 2026-06-04 Thu 06:05:41:175.617 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5e50 2026-06-04 Thu 06:05:41:175.665 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd6930 2026-06-04 Thu 06:05:41:175.737 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7410 2026-06-04 Thu 06:05:41:175.790 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7ef0 2026-06-04 Thu 06:05:41:175.833 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd89d0 2026-06-04 Thu 06:05:41:175.865 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd94b0 2026-06-04 Thu 06:05:41:175.891 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd9f90 2026-06-04 Thu 06:05:41:175.920 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbdaa70 2026-06-04 Thu 06:05:41:175.958 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd4770 2026-06-04 Thu 06:05:41:175.964 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:05:41:175.980 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:05:41:175.989 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd5290 2026-06-04 Thu 06:05:41:175.993 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:05:41:176.010 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:05:41:176.018 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd5e50 2026-06-04 Thu 06:05:41:176.022 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:05:41:176.028 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:05:41:176.034 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd6930 2026-06-04 Thu 06:05:41:176.039 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:05:41:176.045 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:05:41:176.051 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd7410 2026-06-04 Thu 06:05:41:176.055 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:05:41:176.061 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:05:41:176.962 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd89d0 2026-06-04 Thu 06:05:41:176.975 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:05:41:176.985 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:05:41:176.993 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd7ef0 2026-06-04 Thu 06:05:41:176.998 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:05:41:177.005 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:05:41:177.012 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd94b0 2026-06-04 Thu 06:05:41:177.017 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:05:41:177.023 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:05:41:177.030 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd9f90 2026-06-04 Thu 06:05:41:177.034 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:05:41:177.041 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:05:41:177.048 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbdaa70 2026-06-04 Thu 06:05:41:177.058 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:05:41:177.065 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:05:41:177.501 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:05:41:177.526 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:05:41:177.543 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:05:41:177.554 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:05:41:177.565 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:05:41:177.573 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:05:41:177.586 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:05:41:177.594 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:05:41:177.604 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:05:41:177.612 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:05:41:177.622 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:05:41:177.630 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:05:41:177.652 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:05:41:177.699 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:05:41:177.713 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:05:41:177.724 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:05:41:177.743 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:05:41:177.760 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:05:41:177.772 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:05:41:177.780 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:05:41:177.790 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:05:41:177.798 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:05:41:177.808 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:05:41:177.816 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:05:41:177.910 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:05:41:177.920 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:05:41:177.986 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:05:41:178.000 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:05:41:178.011 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:05:41:178.021 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:06:36:093.015 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-06-04 Thu 06:07:36:093.908 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-06-04 Thu 06:07:43:976.003 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Disconnect status: -1 0xbd7410 2026-06-04 Thu 06:07:43:976.059 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-04 Thu 06:07:43:976.251 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Disconnect status: -1 0xbd6930 2026-06-04 Thu 06:07:43:976.274 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-04 Thu 06:07:43:976.313 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Disconnect status: -1 0xbd5e50 2026-06-04 Thu 06:07:43:976.327 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-04 Thu 06:07:43:976.350 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Disconnect status: -1 0xbd5290 2026-06-04 Thu 06:07:43:976.358 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-04 Thu 06:07:43:976.381 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-06-04 Thu 06:07:43:976.387 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Disconnect status: -1 0xbd4770 2026-06-04 Thu 06:07:43:976.394 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-04 Thu 06:07:43:976.575 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7410 2026-06-04 Thu 06:07:43:976.599 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd6930 2026-06-04 Thu 06:07:43:976.610 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5e50 2026-06-04 Thu 06:07:43:976.621 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5290 2026-06-04 Thu 06:07:43:976.637 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd4770 2026-06-04 Thu 06:07:44:034.610 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Disconnect status: -1 0xbdaa70 2026-06-04 Thu 06:07:44:034.662 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-04 Thu 06:07:44:034.714 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Disconnect status: -1 0xbd9f90 2026-06-04 Thu 06:07:44:034.725 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-04 Thu 06:07:44:034.866 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-06-04 Thu 06:07:44:034.876 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Disconnect status: -1 0xbd7ef0 2026-06-04 Thu 06:07:44:034.884 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-04 Thu 06:07:44:034.915 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Disconnect status: -1 0xbd94b0 2026-06-04 Thu 06:07:44:034.938 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-04 Thu 06:07:44:034.971 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Disconnect status: -1 0xbd89d0 2026-06-04 Thu 06:07:44:034.981 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-04 Thu 06:07:44:035.192 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbdaa70 2026-06-04 Thu 06:07:44:035.213 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd9f90 2026-06-04 Thu 06:07:44:035.406 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7ef0 2026-06-04 Thu 06:07:44:035.432 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd94b0 2026-06-04 Thu 06:07:44:035.500 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd89d0 2026-06-04 Thu 06:07:44:173.878 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Disconnect status: -1 0xbd3c50 2026-06-04 Thu 06:07:44:173.939 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-04 Thu 06:07:44:174.064 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Disconnect status: -1 0xbd2830 2026-06-04 Thu 06:07:44:174.077 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-04 Thu 06:07:44:174.334 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Disconnect status: -1 0xbd2180 2026-06-04 Thu 06:07:44:174.343 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-04 Thu 06:07:44:174.366 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-06-04 Thu 06:07:44:174.371 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Disconnect status: -1 0xbcd180 2026-06-04 Thu 06:07:44:174.399 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-04 Thu 06:07:44:174.440 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Disconnect status: -1 0xbd3130 2026-06-04 Thu 06:07:44:174.447 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-06-04 Thu 06:07:44:174.671 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3c50 2026-06-04 Thu 06:07:44:174.821 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2830 2026-06-04 Thu 06:07:44:174.889 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2180 2026-06-04 Thu 06:07:44:174.902 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbcd180 2026-06-04 Thu 06:07:44:174.913 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3130 2026-06-04 Thu 06:07:44:198.241 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:57004::10.0.0.38:8086(-1) OnSessionEvent TCP Connection Closed 2026-06-04 Thu 06:07:44:198.269 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:57004::10.0.0.38:8086(-1):EvTcpClose 2026-06-04 Thu 06:07:44:210.020 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvTcpClose in state Established 2026-06-04 Thu 06:07:44:210.058 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Established : EvTcpClose 2026-06-04 Thu 06:07:44:210.415 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-06-04 Thu 06:07:44:210.508 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:07:44:210.669 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: SANDESH: Send FAILED: 1780553264210165 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 2 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = cn-jenkins-deploy-platform-ansible-os-5888-1. collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155643 calls = 21 average_bytes = 7411 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 22, _iter59->first = sent _iter59->second = 23, ] ] ] 2026-06-04 Thu 06:07:44:210.695 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:07:44:210.710 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:07:44:210.732 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: SANDESH: Send FAILED: 1780553264210207 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 06:07:44:210.740 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:07:44:210.749 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2026-06-04 Thu 06:07:44:210.781 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:07:44:210.906 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: SANDESH: Send FAILED: 1780553264210486 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155643 calls = 21 average_bytes = 7411 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 22, _iter59->first = sent _iter59->second = 23, ] ] ] 2026-06-04 Thu 06:07:44:210.924 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:07:44:211.856 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:34464::10.0.0.50:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 281 2026-06-04 Thu 06:07:44:211.885 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:34464::10.0.0.50:8086(-1) OnSessionEvent TCP Connected 2026-06-04 Thu 06:07:44:211.892 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:34464::10.0.0.50:8086(-1):EvTcpConnected 2026-06-04 Thu 06:07:44:211.913 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-06-04 Thu 06:07:44:211.932 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-06-04 Thu 06:07:44:211.967 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: ClientInit 2026-06-04 Thu 06:07:44:211.986 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5888-2.:contrail-query-engine:0:Database count 3 2026-06-04 Thu 06:07:44:212.059 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:07:44:212.068 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-04 Thu 06:07:44:212.102 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: SANDESH: Send FAILED: 1780553264211961 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 06:07:44:212.111 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-04 Thu 06:07:44:212.120 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:07:44:212.127 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-06-04 Thu 06:07:44:212.137 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:07:44:212.144 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-04 Thu 06:07:44:212.288 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: SANDESH: Send FAILED: 1780553264212045 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155643 calls = 21 average_bytes = 7411 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 25, _iter59->first = sent _iter59->second = 23, ] ] ] 2026-06-04 Thu 06:07:44:212.311 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568021677632, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-04 Thu 06:07:44:253.414 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:34464::10.0.0.50:8086(-1) OnSessionEvent TCP Connection Closed 2026-06-04 Thu 06:07:44:253.437 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:34464::10.0.0.50:8086(-1):EvTcpClose 2026-06-04 Thu 06:07:44:253.509 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-06-04 Thu 06:07:44:253.544 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: ClientInit : EvTcpClose 2026-06-04 Thu 06:07:44:253.696 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: Return to idle with 1000 2026-06-04 Thu 06:07:44:253.707 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: Idle 2026-06-04 Thu 06:07:44:253.767 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:07:44:254.690 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: SANDESH: Send FAILED: 1780553264253637 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 157216 calls = 22 average_bytes = 7146 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 27, _iter59->first = sent _iter59->second = 24, ] ] ] 2026-06-04 Thu 06:07:44:254.741 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:07:44:254.758 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:07:44:254.785 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: SANDESH: Send FAILED: 1780553264253680 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 06:07:44:254.793 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:07:44:254.803 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-04 Thu 06:07:44:254.846 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:07:44:254.985 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: SANDESH: Send FAILED: 1780553264253754 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 157216 calls = 22 average_bytes = 7146 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 27, _iter59->first = sent _iter59->second = 24, ] ] ] 2026-06-04 Thu 06:07:44:255.007 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:07:45:254.249 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568017479232, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-06-04 Thu 06:07:45:254.511 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568017479232, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-06-04 Thu 06:07:45:254.618 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568017479232, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:07:45:254.642 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568017479232, Pid 1]: SANDESH: Send FAILED: 1780553265254325 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 06:07:45:254.650 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568017479232, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:07:45:254.657 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568017479232, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:07:45:254.772 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568017479232, Pid 1]: SANDESH: Send FAILED: 1780553265254603 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 157216 calls = 22 average_bytes = 7146 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 24, ] ] ] 2026-06-04 Thu 06:07:45:254.790 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568017479232, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:07:45:257.380 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:46486::10.0.0.33:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 281 2026-06-04 Thu 06:07:45:257.417 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:46486::10.0.0.33:8086(-1) OnSessionEvent TCP Connected 2026-06-04 Thu 06:07:45:257.428 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:46486::10.0.0.33:8086(-1):EvTcpConnected 2026-06-04 Thu 06:07:45:257.478 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-06-04 Thu 06:07:45:257.499 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-06-04 Thu 06:07:45:257.698 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: ClientInit 2026-06-04 Thu 06:07:45:257.728 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5888-2.:contrail-query-engine:0:Database count 4 2026-06-04 Thu 06:07:45:257.800 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:07:45:257.817 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-04 Thu 06:07:45:257.852 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: SANDESH: Send FAILED: 1780553265257692 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 06:07:45:257.859 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-04 Thu 06:07:45:257.866 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:07:45:257.872 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-06-04 Thu 06:07:45:257.889 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:07:45:257.894 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-04 Thu 06:07:45:257.980 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: SANDESH: Send FAILED: 1780553265257787 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 157216 calls = 22 average_bytes = 7146 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 32, _iter59->first = sent _iter59->second = 24, ] ] ] 2026-06-04 Thu 06:07:45:257.996 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-04 Thu 06:07:45:258.949 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:46486::10.0.0.33:8086(-1) OnSessionEvent TCP Connection Closed 2026-06-04 Thu 06:07:45:258.972 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:46486::10.0.0.33:8086(-1):EvTcpClose 2026-06-04 Thu 06:07:45:259.484 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-06-04 Thu 06:07:45:259.505 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: ClientInit : EvTcpClose 2026-06-04 Thu 06:07:45:259.599 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Return to idle with 1000 2026-06-04 Thu 06:07:45:259.607 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Idle 2026-06-04 Thu 06:07:45:259.646 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:07:45:259.742 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: SANDESH: Send FAILED: 1780553265259562 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 158789 calls = 23 average_bytes = 6903 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 34, _iter59->first = sent _iter59->second = 25, ] ] ] 2026-06-04 Thu 06:07:45:259.760 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:07:45:259.771 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:07:45:259.792 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: SANDESH: Send FAILED: 1780553265259586 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 06:07:45:259.799 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:07:45:259.808 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-04 Thu 06:07:45:259.843 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:07:45:259.920 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: SANDESH: Send FAILED: 1780553265259637 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 158789 calls = 23 average_bytes = 6903 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 34, _iter59->first = sent _iter59->second = 25, ] ] ] 2026-06-04 Thu 06:07:45:259.934 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:07:46:259.774 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-06-04 Thu 06:07:46:260.236 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-06-04 Thu 06:07:46:260.377 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:07:46:260.414 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: SANDESH: Send FAILED: 1780553266259895 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 06:07:46:260.425 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:07:46:260.435 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:07:46:260.670 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: SANDESH: Send FAILED: 1780553266260356 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 158789 calls = 23 average_bytes = 6903 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 37, _iter59->first = sent _iter59->second = 25, ] ] ] 2026-06-04 Thu 06:07:46:260.711 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:07:46:263.103 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:44854::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 281 2026-06-04 Thu 06:07:46:263.145 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:44854::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2026-06-04 Thu 06:07:46:263.155 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:44854::10.0.0.38:8086(-1):EvTcpConnected 2026-06-04 Thu 06:07:46:263.230 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-06-04 Thu 06:07:46:263.269 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-06-04 Thu 06:07:46:263.318 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: ClientInit 2026-06-04 Thu 06:07:46:263.343 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5888-2.:contrail-query-engine:0:Database count 5 2026-06-04 Thu 06:07:46:263.588 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:07:46:263.603 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-04 Thu 06:07:46:263.624 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: SANDESH: Send FAILED: 1780553266263308 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 06:07:46:263.632 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-04 Thu 06:07:46:263.640 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:07:46:263.645 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-06-04 Thu 06:07:46:263.660 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:07:46:263.682 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-04 Thu 06:07:46:263.780 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: SANDESH: Send FAILED: 1780553266263572 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 158789 calls = 23 average_bytes = 6903 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 39, _iter59->first = sent _iter59->second = 25, ] ] ] 2026-06-04 Thu 06:07:46:263.797 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-04 Thu 06:07:48:976.679 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7410 2026-06-04 Thu 06:07:48:976.895 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd6930 2026-06-04 Thu 06:07:48:976.934 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5e50 2026-06-04 Thu 06:07:48:976.963 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5290 2026-06-04 Thu 06:07:48:977.013 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7410 2026-06-04 Thu 06:07:48:977.045 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd6930 2026-06-04 Thu 06:07:48:977.057 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5e50 2026-06-04 Thu 06:07:48:977.135 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5290 2026-06-04 Thu 06:07:48:977.148 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd4770 2026-06-04 Thu 06:07:48:977.189 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd4770 2026-06-04 Thu 06:07:49:035.295 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbdaa70 2026-06-04 Thu 06:07:49:035.535 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd9f90 2026-06-04 Thu 06:07:49:035.585 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7ef0 2026-06-04 Thu 06:07:49:035.625 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd94b0 2026-06-04 Thu 06:07:49:035.663 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd89d0 2026-06-04 Thu 06:07:49:036.178 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbdaa70 2026-06-04 Thu 06:07:49:036.245 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd9f90 2026-06-04 Thu 06:07:49:036.265 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd94b0 2026-06-04 Thu 06:07:49:036.282 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd89d0 2026-06-04 Thu 06:07:49:039.145 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7ef0 2026-06-04 Thu 06:07:49:174.754 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3c50 2026-06-04 Thu 06:07:49:174.983 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2830 2026-06-04 Thu 06:07:49:175.026 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2180 2026-06-04 Thu 06:07:49:175.055 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbcd180 2026-06-04 Thu 06:07:49:175.081 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3130 2026-06-04 Thu 06:07:49:175.352 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3c50 2026-06-04 Thu 06:07:49:175.399 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2830 2026-06-04 Thu 06:07:49:175.412 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2180 2026-06-04 Thu 06:07:49:175.423 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3130 2026-06-04 Thu 06:07:49:175.435 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbcd180 2026-06-04 Thu 06:07:50:263.451 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567979705920, Pid 1]: 10.0.0.38:8086 EvConnectTimerExpired in state ClientInit 2026-06-04 Thu 06:07:50:263.567 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Processing scm::EvConnectTimerExpired in state ClientInit 2026-06-04 Thu 06:07:50:263.604 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: ClientInit : EvConnectTimerExpired 2026-06-04 Thu 06:07:50:263.811 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Return to idle with 1000 2026-06-04 Thu 06:07:50:263.823 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Idle 2026-06-04 Thu 06:07:50:263.876 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:07:50:263.984 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: SANDESH: Send FAILED: 1780553270263699 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 160362 calls = 24 average_bytes = 6681 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 41, _iter59->first = sent _iter59->second = 26, ] ] ] 2026-06-04 Thu 06:07:50:264.008 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:07:50:264.018 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:07:50:264.035 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: SANDESH: Send FAILED: 1780553270263735 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Down description = ClientInit : EvConnectTimerExpired ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 06:07:50:264.044 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:07:50:264.051 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-04 Thu 06:07:50:264.102 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:07:50:264.186 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: SANDESH: Send FAILED: 1780553270263864 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 160362 calls = 24 average_bytes = 6681 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 41, _iter59->first = sent _iter59->second = 26, ] ] ] 2026-06-04 Thu 06:07:50:264.201 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:07:51:264.116 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-06-04 Thu 06:07:51:264.388 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-06-04 Thu 06:07:51:264.481 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:07:51:264.504 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: SANDESH: Send FAILED: 1780553271264190 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 06:07:51:264.518 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:07:51:264.526 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:07:51:264.667 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: SANDESH: Send FAILED: 1780553271264465 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 160362 calls = 24 average_bytes = 6681 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 44, _iter59->first = sent _iter59->second = 26, ] ] ] 2026-06-04 Thu 06:07:51:264.685 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:07:51:266.681 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:59220::10.0.0.50:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 281 2026-06-04 Thu 06:07:51:266.756 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:59220::10.0.0.50:8086(-1) OnSessionEvent TCP Connected 2026-06-04 Thu 06:07:51:266.785 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:59220::10.0.0.50:8086(-1):EvTcpConnected 2026-06-04 Thu 06:07:51:266.874 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-06-04 Thu 06:07:51:266.907 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-06-04 Thu 06:07:51:266.948 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: ClientInit 2026-06-04 Thu 06:07:51:266.970 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5888-2.:contrail-query-engine:0:Database count 6 2026-06-04 Thu 06:07:51:267.082 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:07:51:267.102 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-04 Thu 06:07:51:267.123 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: SANDESH: Send FAILED: 1780553271266940 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 06:07:51:267.131 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-04 Thu 06:07:51:267.141 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:07:51:267.149 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-06-04 Thu 06:07:51:267.162 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:07:51:267.170 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-04 Thu 06:07:51:267.261 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: SANDESH: Send FAILED: 1780553271267064 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 160362 calls = 24 average_bytes = 6681 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 46, _iter59->first = sent _iter59->second = 26, ] ] ] 2026-06-04 Thu 06:07:51:267.282 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-04 Thu 06:07:53:977.220 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7410 2026-06-04 Thu 06:07:53:977.489 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd6930 2026-06-04 Thu 06:07:53:977.580 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7410 2026-06-04 Thu 06:07:53:977.615 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd6930 2026-06-04 Thu 06:07:53:977.626 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5e50 2026-06-04 Thu 06:07:53:977.657 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5290 2026-06-04 Thu 06:07:53:977.684 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd4770 2026-06-04 Thu 06:07:53:977.714 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5e50 2026-06-04 Thu 06:07:53:977.727 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5290 2026-06-04 Thu 06:07:53:977.738 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd4770 2026-06-04 Thu 06:07:54:036.285 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbdaa70 2026-06-04 Thu 06:07:54:036.543 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd9f90 2026-06-04 Thu 06:07:54:036.611 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd94b0 2026-06-04 Thu 06:07:54:036.664 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd89d0 2026-06-04 Thu 06:07:54:036.916 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbdaa70 2026-06-04 Thu 06:07:54:037.003 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd9f90 2026-06-04 Thu 06:07:54:037.026 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd94b0 2026-06-04 Thu 06:07:54:037.043 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd89d0 2026-06-04 Thu 06:07:54:039.246 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7ef0 2026-06-04 Thu 06:07:54:039.709 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7ef0 2026-06-04 Thu 06:07:54:175.453 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3c50 2026-06-04 Thu 06:07:54:175.654 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2830 2026-06-04 Thu 06:07:54:175.693 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2180 2026-06-04 Thu 06:07:54:175.720 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3130 2026-06-04 Thu 06:07:54:175.743 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbcd180 2026-06-04 Thu 06:07:54:176.097 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3c50 2026-06-04 Thu 06:07:54:176.132 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2830 2026-06-04 Thu 06:07:54:176.145 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3130 2026-06-04 Thu 06:07:54:176.156 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2180 2026-06-04 Thu 06:07:54:176.169 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbcd180 2026-06-04 Thu 06:07:56:041.368 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.50:59220::10.0.0.50:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 506 2026-06-04 Thu 06:07:56:041.424 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.50:59220::10.0.0.50:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2026-06-04 Thu 06:07:56:041.453 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:59220::10.0.0.50:8086(-1) OnSessionEvent TCP Connection Closed 2026-06-04 Thu 06:07:56:041.464 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:59220::10.0.0.50:8086(-1):EvTcpClose 2026-06-04 Thu 06:07:56:041.554 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-06-04 Thu 06:07:56:041.592 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: ClientInit : EvTcpClose 2026-06-04 Thu 06:07:56:041.757 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Return to idle with 1000 2026-06-04 Thu 06:07:56:041.771 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Idle 2026-06-04 Thu 06:07:56:041.885 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:07:56:042.063 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: SANDESH: Send FAILED: 1780553276041691 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 48, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-06-04 Thu 06:07:56:042.123 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:07:56:042.139 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:07:56:042.165 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: SANDESH: Send FAILED: 1780553276041734 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 06:07:56:042.174 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:07:56:042.185 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-04 Thu 06:07:56:042.258 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:07:56:042.407 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: SANDESH: Send FAILED: 1780553276041824 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 48, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-06-04 Thu 06:07:56:042.429 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:07:57:041.972 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-06-04 Thu 06:07:57:042.319 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-06-04 Thu 06:07:57:042.437 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:07:57:042.469 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: SANDESH: Send FAILED: 1780553277042077 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 06:07:57:042.491 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:07:57:042.502 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:07:57:042.668 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: SANDESH: Send FAILED: 1780553277042418 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 51, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-06-04 Thu 06:07:57:042.691 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:07:57:042.781 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-06-04 Thu 06:07:57:042.832 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: -(-1):EvTcpConnectFail 2026-06-04 Thu 06:07:57:042.897 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-06-04 Thu 06:07:57:042.934 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: Connect : EvTcpConnectFail 2026-06-04 Thu 06:07:57:043.063 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2026-06-04 Thu 06:07:57:043.117 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: Idle 2026-06-04 Thu 06:07:57:043.181 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:07:57:043.280 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: SANDESH: Send FAILED: 1780553277043010 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 53, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-06-04 Thu 06:07:57:043.301 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:07:57:043.312 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:07:57:043.336 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: SANDESH: Send FAILED: 1780553277043039 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 06:07:57:043.347 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:07:57:043.356 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-04 Thu 06:07:57:043.392 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:07:57:043.521 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: SANDESH: Send FAILED: 1780553277043162 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 53, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-06-04 Thu 06:07:57:043.536 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:07:58:977.658 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7410 2026-06-04 Thu 06:07:58:977.797 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd6930 2026-06-04 Thu 06:07:58:977.861 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7410 2026-06-04 Thu 06:07:58:977.886 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd6930 2026-06-04 Thu 06:07:58:977.895 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5e50 2026-06-04 Thu 06:07:58:977.924 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5290 2026-06-04 Thu 06:07:58:977.958 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd4770 2026-06-04 Thu 06:07:58:977.987 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5e50 2026-06-04 Thu 06:07:58:978.000 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5290 2026-06-04 Thu 06:07:58:978.011 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd4770 2026-06-04 Thu 06:07:59:037.039 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbdaa70 2026-06-04 Thu 06:07:59:037.231 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd9f90 2026-06-04 Thu 06:07:59:037.284 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd94b0 2026-06-04 Thu 06:07:59:037.314 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd89d0 2026-06-04 Thu 06:07:59:037.450 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbdaa70 2026-06-04 Thu 06:07:59:037.514 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd94b0 2026-06-04 Thu 06:07:59:037.535 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd9f90 2026-06-04 Thu 06:07:59:037.554 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd89d0 2026-06-04 Thu 06:07:59:039.779 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7ef0 2026-06-04 Thu 06:07:59:040.245 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7ef0 2026-06-04 Thu 06:07:59:176.160 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3c50 2026-06-04 Thu 06:07:59:176.275 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2830 2026-06-04 Thu 06:07:59:176.309 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3130 2026-06-04 Thu 06:07:59:176.338 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2180 2026-06-04 Thu 06:07:59:176.361 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbcd180 2026-06-04 Thu 06:07:59:176.631 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2830 2026-06-04 Thu 06:07:59:176.659 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3130 2026-06-04 Thu 06:07:59:176.671 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2180 2026-06-04 Thu 06:07:59:176.794 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbcd180 2026-06-04 Thu 06:07:59:180.010 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3c50 2026-06-04 Thu 06:08:03:977.934 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7410 2026-06-04 Thu 06:08:03:978.106 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd6930 2026-06-04 Thu 06:08:03:978.166 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7410 2026-06-04 Thu 06:08:03:978.192 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd6930 2026-06-04 Thu 06:08:03:978.201 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5e50 2026-06-04 Thu 06:08:03:978.226 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5290 2026-06-04 Thu 06:08:03:978.251 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd4770 2026-06-04 Thu 06:08:03:978.292 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5e50 2026-06-04 Thu 06:08:03:978.311 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5290 2026-06-04 Thu 06:08:03:978.322 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd4770 2026-06-04 Thu 06:08:04:037.513 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbdaa70 2026-06-04 Thu 06:08:04:037.660 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd94b0 2026-06-04 Thu 06:08:04:037.693 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd9f90 2026-06-04 Thu 06:08:04:037.720 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd89d0 2026-06-04 Thu 06:08:04:038.188 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbdaa70 2026-06-04 Thu 06:08:04:038.214 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd94b0 2026-06-04 Thu 06:08:04:038.226 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd9f90 2026-06-04 Thu 06:08:04:038.237 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd89d0 2026-06-04 Thu 06:08:04:040.300 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7ef0 2026-06-04 Thu 06:08:04:040.822 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7ef0 2026-06-04 Thu 06:08:04:176.724 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2830 2026-06-04 Thu 06:08:04:176.887 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3130 2026-06-04 Thu 06:08:04:176.971 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2180 2026-06-04 Thu 06:08:04:177.010 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbcd180 2026-06-04 Thu 06:08:04:177.673 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2830 2026-06-04 Thu 06:08:04:177.709 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3130 2026-06-04 Thu 06:08:04:177.721 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2180 2026-06-04 Thu 06:08:04:177.767 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbcd180 2026-06-04 Thu 06:08:04:180.062 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3c50 2026-06-04 Thu 06:08:04:180.409 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3c50 2026-06-04 Thu 06:08:06:094.280 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:08:06:094.372 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: SANDESH: Send FAILED: 1780553286094169 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 83 dequeues = 83 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 114 dequeues = 114 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1780553277043116 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1780553277043390 ] max_sm_queue_count = 6 ] 2026-06-04 Thu 06:08:06:094.386 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:08:06:094.394 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:08:06:094.493 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: SANDESH: Send FAILED: 1780553286094255 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 56, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-06-04 Thu 06:08:06:094.515 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:08:08:978.245 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7410 2026-06-04 Thu 06:08:08:978.449 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd6930 2026-06-04 Thu 06:08:08:978.514 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7410 2026-06-04 Thu 06:08:08:978.549 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd6930 2026-06-04 Thu 06:08:08:978.563 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5e50 2026-06-04 Thu 06:08:08:978.603 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5290 2026-06-04 Thu 06:08:08:978.635 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd4770 2026-06-04 Thu 06:08:08:978.673 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5e50 2026-06-04 Thu 06:08:08:978.690 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5290 2026-06-04 Thu 06:08:08:978.704 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd4770 2026-06-04 Thu 06:08:09:038.252 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbdaa70 2026-06-04 Thu 06:08:09:038.395 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd94b0 2026-06-04 Thu 06:08:09:038.431 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd9f90 2026-06-04 Thu 06:08:09:038.458 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd89d0 2026-06-04 Thu 06:08:09:038.767 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbdaa70 2026-06-04 Thu 06:08:09:038.794 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd94b0 2026-06-04 Thu 06:08:09:038.806 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd9f90 2026-06-04 Thu 06:08:09:038.825 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd89d0 2026-06-04 Thu 06:08:09:040.858 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7ef0 2026-06-04 Thu 06:08:09:041.207 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7ef0 2026-06-04 Thu 06:08:09:177.759 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2830 2026-06-04 Thu 06:08:09:177.920 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3130 2026-06-04 Thu 06:08:09:177.959 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2180 2026-06-04 Thu 06:08:09:177.994 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbcd180 2026-06-04 Thu 06:08:09:178.316 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2830 2026-06-04 Thu 06:08:09:178.346 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3130 2026-06-04 Thu 06:08:09:178.358 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2180 2026-06-04 Thu 06:08:09:178.369 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbcd180 2026-06-04 Thu 06:08:09:180.449 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3c50 2026-06-04 Thu 06:08:09:180.911 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3c50 2026-06-04 Thu 06:08:13:043.266 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-06-04 Thu 06:08:13:043.572 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-06-04 Thu 06:08:13:043.679 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:08:13:043.704 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: SANDESH: Send FAILED: 1780553293043370 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 06:08:13:043.711 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:08:13:043.719 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:08:13:043.850 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: SANDESH: Send FAILED: 1780553293043663 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 58, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-06-04 Thu 06:08:13:043.875 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:08:13:045.468 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-06-04 Thu 06:08:13:045.517 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: -(-1):EvTcpConnectFail 2026-06-04 Thu 06:08:13:045.574 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-06-04 Thu 06:08:13:045.595 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Connect : EvTcpConnectFail 2026-06-04 Thu 06:08:13:045.764 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2026-06-04 Thu 06:08:13:045.822 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Idle 2026-06-04 Thu 06:08:13:045.909 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:08:13:046.071 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: SANDESH: Send FAILED: 1780553293045684 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 60, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-06-04 Thu 06:08:13:046.162 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:08:13:046.179 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:08:13:046.198 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: SANDESH: Send FAILED: 1780553293045732 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 06:08:13:046.205 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:08:13:046.212 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-04 Thu 06:08:13:046.248 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:08:13:046.331 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: SANDESH: Send FAILED: 1780553293045895 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 60, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-06-04 Thu 06:08:13:046.350 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567023441472, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:08:13:978.616 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7410 2026-06-04 Thu 06:08:13:978.848 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd6930 2026-06-04 Thu 06:08:13:978.930 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7410 2026-06-04 Thu 06:08:13:978.967 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd6930 2026-06-04 Thu 06:08:13:978.982 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5e50 2026-06-04 Thu 06:08:13:979.024 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5290 2026-06-04 Thu 06:08:13:979.060 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd4770 2026-06-04 Thu 06:08:13:979.157 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5e50 2026-06-04 Thu 06:08:13:979.182 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5290 2026-06-04 Thu 06:08:13:979.197 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd4770 2026-06-04 Thu 06:08:14:038.855 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbdaa70 2026-06-04 Thu 06:08:14:039.027 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd94b0 2026-06-04 Thu 06:08:14:039.090 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd9f90 2026-06-04 Thu 06:08:14:039.130 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd89d0 2026-06-04 Thu 06:08:14:039.437 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbdaa70 2026-06-04 Thu 06:08:14:039.480 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd94b0 2026-06-04 Thu 06:08:14:039.496 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd9f90 2026-06-04 Thu 06:08:14:039.511 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd89d0 2026-06-04 Thu 06:08:14:041.244 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7ef0 2026-06-04 Thu 06:08:14:041.600 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7ef0 2026-06-04 Thu 06:08:14:178.413 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2830 2026-06-04 Thu 06:08:14:178.594 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3130 2026-06-04 Thu 06:08:14:178.641 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2180 2026-06-04 Thu 06:08:14:178.677 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbcd180 2026-06-04 Thu 06:08:14:179.002 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2830 2026-06-04 Thu 06:08:14:179.040 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3130 2026-06-04 Thu 06:08:14:179.056 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbcd180 2026-06-04 Thu 06:08:14:179.072 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2180 2026-06-04 Thu 06:08:14:180.975 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3c50 2026-06-04 Thu 06:08:14:181.397 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3c50 2026-06-04 Thu 06:08:18:979.033 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7410 2026-06-04 Thu 06:08:18:979.262 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd6930 2026-06-04 Thu 06:08:18:979.333 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7410 2026-06-04 Thu 06:08:18:979.365 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd6930 2026-06-04 Thu 06:08:18:979.378 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5e50 2026-06-04 Thu 06:08:18:979.412 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5290 2026-06-04 Thu 06:08:18:979.446 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd4770 2026-06-04 Thu 06:08:18:979.485 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5e50 2026-06-04 Thu 06:08:18:979.514 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5290 2026-06-04 Thu 06:08:18:979.527 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd4770 2026-06-04 Thu 06:08:19:039.525 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbdaa70 2026-06-04 Thu 06:08:19:039.662 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd94b0 2026-06-04 Thu 06:08:19:039.699 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd9f90 2026-06-04 Thu 06:08:19:039.725 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd89d0 2026-06-04 Thu 06:08:19:040.179 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbdaa70 2026-06-04 Thu 06:08:19:040.219 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd94b0 2026-06-04 Thu 06:08:19:040.236 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd9f90 2026-06-04 Thu 06:08:19:041.639 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7ef0 2026-06-04 Thu 06:08:19:042.150 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7ef0 2026-06-04 Thu 06:08:19:043.737 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd89d0 2026-06-04 Thu 06:08:19:179.176 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2830 2026-06-04 Thu 06:08:19:179.425 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3130 2026-06-04 Thu 06:08:19:179.496 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbcd180 2026-06-04 Thu 06:08:19:179.538 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2180 2026-06-04 Thu 06:08:19:179.755 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2830 2026-06-04 Thu 06:08:19:179.819 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3130 2026-06-04 Thu 06:08:19:179.839 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbcd180 2026-06-04 Thu 06:08:19:179.858 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2180 2026-06-04 Thu 06:08:19:181.468 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3c50 2026-06-04 Thu 06:08:19:182.214 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3c50 2026-06-04 Thu 06:08:23:979.429 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7410 2026-06-04 Thu 06:08:23:979.719 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd6930 2026-06-04 Thu 06:08:23:979.777 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7410 2026-06-04 Thu 06:08:23:979.805 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd6930 2026-06-04 Thu 06:08:23:979.829 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5e50 2026-06-04 Thu 06:08:23:979.861 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5290 2026-06-04 Thu 06:08:23:979.892 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd4770 2026-06-04 Thu 06:08:23:979.926 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5e50 2026-06-04 Thu 06:08:23:979.941 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5290 2026-06-04 Thu 06:08:23:979.954 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd4770 2026-06-04 Thu 06:08:24:040.268 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbdaa70 2026-06-04 Thu 06:08:24:040.416 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd94b0 2026-06-04 Thu 06:08:24:040.457 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd9f90 2026-06-04 Thu 06:08:24:040.794 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbdaa70 2026-06-04 Thu 06:08:24:040.825 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd94b0 2026-06-04 Thu 06:08:24:040.839 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd9f90 2026-06-04 Thu 06:08:24:042.187 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7ef0 2026-06-04 Thu 06:08:24:042.642 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7ef0 2026-06-04 Thu 06:08:24:043.806 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd89d0 2026-06-04 Thu 06:08:24:044.263 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd89d0 2026-06-04 Thu 06:08:24:179.872 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2830 2026-06-04 Thu 06:08:24:180.027 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3130 2026-06-04 Thu 06:08:24:180.068 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbcd180 2026-06-04 Thu 06:08:24:180.110 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2180 2026-06-04 Thu 06:08:24:180.390 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2830 2026-06-04 Thu 06:08:24:180.441 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbcd180 2026-06-04 Thu 06:08:24:180.462 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3130 2026-06-04 Thu 06:08:24:180.484 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2180 2026-06-04 Thu 06:08:24:182.296 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3c50 2026-06-04 Thu 06:08:24:182.690 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3c50 2026-06-04 Thu 06:08:28:979.877 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7410 2026-06-04 Thu 06:08:28:980.131 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd6930 2026-06-04 Thu 06:08:28:980.203 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7410 2026-06-04 Thu 06:08:28:980.239 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd6930 2026-06-04 Thu 06:08:28:980.252 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5e50 2026-06-04 Thu 06:08:28:980.291 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5290 2026-06-04 Thu 06:08:28:980.325 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd4770 2026-06-04 Thu 06:08:28:980.365 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5e50 2026-06-04 Thu 06:08:28:980.382 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5290 2026-06-04 Thu 06:08:28:980.397 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd4770 2026-06-04 Thu 06:08:29:040.877 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbdaa70 2026-06-04 Thu 06:08:29:041.041 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd94b0 2026-06-04 Thu 06:08:29:041.098 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd9f90 2026-06-04 Thu 06:08:29:041.476 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbdaa70 2026-06-04 Thu 06:08:29:041.525 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd94b0 2026-06-04 Thu 06:08:29:041.542 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd9f90 2026-06-04 Thu 06:08:29:042.702 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7ef0 2026-06-04 Thu 06:08:29:043.100 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7ef0 2026-06-04 Thu 06:08:29:044.301 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd89d0 2026-06-04 Thu 06:08:29:044.628 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd89d0 2026-06-04 Thu 06:08:29:180.476 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2830 2026-06-04 Thu 06:08:29:180.592 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbcd180 2026-06-04 Thu 06:08:29:180.625 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3130 2026-06-04 Thu 06:08:29:180.650 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2180 2026-06-04 Thu 06:08:29:180.876 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbcd180 2026-06-04 Thu 06:08:29:180.902 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2830 2026-06-04 Thu 06:08:29:180.923 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3130 2026-06-04 Thu 06:08:29:180.935 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2180 2026-06-04 Thu 06:08:29:182.747 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3c50 2026-06-04 Thu 06:08:29:183.095 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3c50 2026-06-04 Thu 06:08:33:980.279 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7410 2026-06-04 Thu 06:08:33:980.422 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd6930 2026-06-04 Thu 06:08:33:980.470 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7410 2026-06-04 Thu 06:08:33:980.493 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd6930 2026-06-04 Thu 06:08:33:980.501 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5e50 2026-06-04 Thu 06:08:33:980.527 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5290 2026-06-04 Thu 06:08:33:980.550 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd4770 2026-06-04 Thu 06:08:33:980.577 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5e50 2026-06-04 Thu 06:08:33:980.589 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5290 2026-06-04 Thu 06:08:33:980.599 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd4770 2026-06-04 Thu 06:08:34:041.576 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbdaa70 2026-06-04 Thu 06:08:34:041.724 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd94b0 2026-06-04 Thu 06:08:34:041.755 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd9f90 2026-06-04 Thu 06:08:34:042.145 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbdaa70 2026-06-04 Thu 06:08:34:042.173 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd9f90 2026-06-04 Thu 06:08:34:042.186 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd94b0 2026-06-04 Thu 06:08:34:043.150 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7ef0 2026-06-04 Thu 06:08:34:043.482 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7ef0 2026-06-04 Thu 06:08:34:044.660 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd89d0 2026-06-04 Thu 06:08:34:045.031 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd89d0 2026-06-04 Thu 06:08:34:180.963 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbcd180 2026-06-04 Thu 06:08:34:181.117 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2830 2026-06-04 Thu 06:08:34:181.179 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3130 2026-06-04 Thu 06:08:34:181.221 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2180 2026-06-04 Thu 06:08:34:181.474 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3130 2026-06-04 Thu 06:08:34:181.512 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbcd180 2026-06-04 Thu 06:08:34:181.530 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2180 2026-06-04 Thu 06:08:34:181.547 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2830 2026-06-04 Thu 06:08:34:183.162 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3c50 2026-06-04 Thu 06:08:34:183.493 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3c50 2026-06-04 Thu 06:08:36:094.622 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-06-04 Thu 06:08:36:094.686 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:08:36:094.732 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SANDESH: Send FAILED: 1780553316094399 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 90 dequeues = 90 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 124 dequeues = 124 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1780553293045821 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1780553293046247 ] max_sm_queue_count = 6 ] 2026-06-04 Thu 06:08:36:094.744 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:08:36:094.752 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:08:36:094.936 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SANDESH: Send FAILED: 1780553316094604 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 63, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-06-04 Thu 06:08:36:094.958 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:08:38:980.548 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7410 2026-06-04 Thu 06:08:38:980.725 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd6930 2026-06-04 Thu 06:08:38:980.775 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7410 2026-06-04 Thu 06:08:38:980.799 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd6930 2026-06-04 Thu 06:08:38:980.808 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5e50 2026-06-04 Thu 06:08:38:980.832 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5290 2026-06-04 Thu 06:08:38:980.855 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd4770 2026-06-04 Thu 06:08:38:980.882 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5e50 2026-06-04 Thu 06:08:38:980.894 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5290 2026-06-04 Thu 06:08:38:980.904 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd4770 2026-06-04 Thu 06:08:39:042.213 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbdaa70 2026-06-04 Thu 06:08:39:042.348 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd9f90 2026-06-04 Thu 06:08:39:042.399 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd94b0 2026-06-04 Thu 06:08:39:042.735 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbdaa70 2026-06-04 Thu 06:08:39:042.765 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd94b0 2026-06-04 Thu 06:08:39:042.778 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd9f90 2026-06-04 Thu 06:08:39:043.508 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7ef0 2026-06-04 Thu 06:08:39:043.830 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7ef0 2026-06-04 Thu 06:08:39:045.093 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd89d0 2026-06-04 Thu 06:08:39:045.590 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd89d0 2026-06-04 Thu 06:08:39:181.552 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3130 2026-06-04 Thu 06:08:39:181.681 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbcd180 2026-06-04 Thu 06:08:39:181.716 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2180 2026-06-04 Thu 06:08:39:181.742 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2830 2026-06-04 Thu 06:08:39:182.093 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2830 2026-06-04 Thu 06:08:39:182.141 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2180 2026-06-04 Thu 06:08:39:182.161 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbcd180 2026-06-04 Thu 06:08:39:182.174 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3130 2026-06-04 Thu 06:08:39:183.628 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3c50 2026-06-04 Thu 06:08:39:184.110 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3c50 2026-06-04 Thu 06:08:41:176.838 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567027639872, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:08:41:176.939 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567027639872, Pid 1]: SANDESH: Send FAILED: 1780553321176801 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2026-06-04 Thu 06:08:41:176.951 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567027639872, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2026-06-04 Thu 06:08:43:046.216 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-06-04 Thu 06:08:43:046.452 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-06-04 Thu 06:08:43:046.533 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:08:43:046.555 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SANDESH: Send FAILED: 1780553323046302 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 06:08:43:046.563 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:08:43:046.571 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:08:43:046.663 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SANDESH: Send FAILED: 1780553323046521 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 66, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-06-04 Thu 06:08:43:046.681 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:08:43:047.235 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-06-04 Thu 06:08:43:047.243 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: -(-1):EvTcpConnectFail 2026-06-04 Thu 06:08:43:047.732 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-06-04 Thu 06:08:43:047.739 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Connect : EvTcpConnectFail 2026-06-04 Thu 06:08:43:047.819 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2026-06-04 Thu 06:08:43:047.848 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Idle 2026-06-04 Thu 06:08:43:047.889 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:08:43:047.972 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SANDESH: Send FAILED: 1780553323047777 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 68, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-06-04 Thu 06:08:43:047.987 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:08:43:047.997 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:08:43:048.012 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SANDESH: Send FAILED: 1780553323047801 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 06:08:43:048.017 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:08:43:048.023 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-04 Thu 06:08:43:048.030 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:08:43:048.111 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SANDESH: Send FAILED: 1780553323047880 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 68, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-06-04 Thu 06:08:43:048.126 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:08:43:980.837 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7410 2026-06-04 Thu 06:08:43:981.004 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd6930 2026-06-04 Thu 06:08:43:981.052 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7410 2026-06-04 Thu 06:08:43:981.086 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd6930 2026-06-04 Thu 06:08:43:981.095 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5e50 2026-06-04 Thu 06:08:43:981.122 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5290 2026-06-04 Thu 06:08:43:981.145 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd4770 2026-06-04 Thu 06:08:43:981.171 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5e50 2026-06-04 Thu 06:08:43:981.184 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5290 2026-06-04 Thu 06:08:43:981.194 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd4770 2026-06-04 Thu 06:08:44:042.798 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbdaa70 2026-06-04 Thu 06:08:44:042.914 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd94b0 2026-06-04 Thu 06:08:44:042.945 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd9f90 2026-06-04 Thu 06:08:44:043.252 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbdaa70 2026-06-04 Thu 06:08:44:043.293 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd94b0 2026-06-04 Thu 06:08:44:043.305 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd9f90 2026-06-04 Thu 06:08:44:043.855 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7ef0 2026-06-04 Thu 06:08:44:044.218 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7ef0 2026-06-04 Thu 06:08:44:045.635 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd89d0 2026-06-04 Thu 06:08:44:046.034 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd89d0 2026-06-04 Thu 06:08:44:182.197 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2830 2026-06-04 Thu 06:08:44:182.364 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2180 2026-06-04 Thu 06:08:44:182.401 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbcd180 2026-06-04 Thu 06:08:44:182.425 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3130 2026-06-04 Thu 06:08:44:182.771 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2830 2026-06-04 Thu 06:08:44:182.799 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2180 2026-06-04 Thu 06:08:44:182.813 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbcd180 2026-06-04 Thu 06:08:44:182.824 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3130 2026-06-04 Thu 06:08:44:184.178 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3c50 2026-06-04 Thu 06:08:44:184.620 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3c50 2026-06-04 Thu 06:08:48:981.138 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7410 2026-06-04 Thu 06:08:48:981.349 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd6930 2026-06-04 Thu 06:08:48:981.412 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7410 2026-06-04 Thu 06:08:48:981.465 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd6930 2026-06-04 Thu 06:08:48:981.480 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5e50 2026-06-04 Thu 06:08:48:981.516 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5290 2026-06-04 Thu 06:08:48:981.550 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd4770 2026-06-04 Thu 06:08:48:981.591 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5e50 2026-06-04 Thu 06:08:48:981.608 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5290 2026-06-04 Thu 06:08:48:981.623 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd4770 2026-06-04 Thu 06:08:49:043.323 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbdaa70 2026-06-04 Thu 06:08:49:043.434 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd94b0 2026-06-04 Thu 06:08:49:043.465 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd9f90 2026-06-04 Thu 06:08:49:044.478 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd94b0 2026-06-04 Thu 06:08:49:044.522 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd9f90 2026-06-04 Thu 06:08:49:044.535 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbdaa70 2026-06-04 Thu 06:08:49:044.544 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7ef0 2026-06-04 Thu 06:08:49:045.084 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7ef0 2026-06-04 Thu 06:08:49:046.086 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd89d0 2026-06-04 Thu 06:08:49:046.444 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd89d0 2026-06-04 Thu 06:08:49:182.840 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2830 2026-06-04 Thu 06:08:49:182.958 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2180 2026-06-04 Thu 06:08:49:182.989 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbcd180 2026-06-04 Thu 06:08:49:183.010 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3130 2026-06-04 Thu 06:08:49:183.552 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2830 2026-06-04 Thu 06:08:49:183.583 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2180 2026-06-04 Thu 06:08:49:183.595 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbcd180 2026-06-04 Thu 06:08:49:183.607 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3130 2026-06-04 Thu 06:08:49:184.660 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3c50 2026-06-04 Thu 06:08:49:185.008 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3c50 2026-06-04 Thu 06:08:53:981.506 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7410 2026-06-04 Thu 06:08:53:981.661 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd6930 2026-06-04 Thu 06:08:53:981.712 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7410 2026-06-04 Thu 06:08:53:981.737 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd6930 2026-06-04 Thu 06:08:53:981.746 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5e50 2026-06-04 Thu 06:08:53:981.773 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5290 2026-06-04 Thu 06:08:53:981.795 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd4770 2026-06-04 Thu 06:08:53:981.824 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5e50 2026-06-04 Thu 06:08:53:981.880 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5290 2026-06-04 Thu 06:08:53:981.892 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd4770 2026-06-04 Thu 06:08:54:044.554 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd94b0 2026-06-04 Thu 06:08:54:044.664 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd9f90 2026-06-04 Thu 06:08:54:044.696 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbdaa70 2026-06-04 Thu 06:08:54:045.096 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd94b0 2026-06-04 Thu 06:08:54:045.146 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd9f90 2026-06-04 Thu 06:08:54:045.166 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbdaa70 2026-06-04 Thu 06:08:54:045.191 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7ef0 2026-06-04 Thu 06:08:54:045.604 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7ef0 2026-06-04 Thu 06:08:54:046.494 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd89d0 2026-06-04 Thu 06:08:54:046.859 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd89d0 2026-06-04 Thu 06:08:54:183.627 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2830 2026-06-04 Thu 06:08:54:183.744 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2180 2026-06-04 Thu 06:08:54:183.781 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbcd180 2026-06-04 Thu 06:08:54:183.806 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3130 2026-06-04 Thu 06:08:54:184.057 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbcd180 2026-06-04 Thu 06:08:54:184.091 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2180 2026-06-04 Thu 06:08:54:184.112 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2830 2026-06-04 Thu 06:08:54:184.124 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3130 2026-06-04 Thu 06:08:54:185.042 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3c50 2026-06-04 Thu 06:08:54:185.392 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3c50 2026-06-04 Thu 06:08:58:981.794 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7410 2026-06-04 Thu 06:08:58:981.950 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd6930 2026-06-04 Thu 06:08:58:982.016 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7410 2026-06-04 Thu 06:08:58:982.040 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd6930 2026-06-04 Thu 06:08:58:982.049 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5e50 2026-06-04 Thu 06:08:58:982.089 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5290 2026-06-04 Thu 06:08:58:982.113 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd4770 2026-06-04 Thu 06:08:58:982.142 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5e50 2026-06-04 Thu 06:08:58:982.154 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5290 2026-06-04 Thu 06:08:58:982.164 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd4770 2026-06-04 Thu 06:08:59:045.200 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd94b0 2026-06-04 Thu 06:08:59:045.328 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd9f90 2026-06-04 Thu 06:08:59:045.379 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbdaa70 2026-06-04 Thu 06:08:59:045.648 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7ef0 2026-06-04 Thu 06:08:59:045.707 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd9f90 2026-06-04 Thu 06:08:59:045.741 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd94b0 2026-06-04 Thu 06:08:59:045.754 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbdaa70 2026-06-04 Thu 06:08:59:045.985 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7ef0 2026-06-04 Thu 06:08:59:046.907 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd89d0 2026-06-04 Thu 06:08:59:047.279 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd89d0 2026-06-04 Thu 06:08:59:184.128 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbcd180 2026-06-04 Thu 06:08:59:184.253 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2180 2026-06-04 Thu 06:08:59:184.289 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2830 2026-06-04 Thu 06:08:59:184.318 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3130 2026-06-04 Thu 06:08:59:184.548 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbcd180 2026-06-04 Thu 06:08:59:184.608 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2180 2026-06-04 Thu 06:08:59:184.630 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3130 2026-06-04 Thu 06:08:59:185.131 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2830 2026-06-04 Thu 06:08:59:185.423 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3c50 2026-06-04 Thu 06:08:59:185.949 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3c50 2026-06-04 Thu 06:09:03:982.129 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7410 2026-06-04 Thu 06:09:03:982.322 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd6930 2026-06-04 Thu 06:09:03:982.384 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7410 2026-06-04 Thu 06:09:03:982.413 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd6930 2026-06-04 Thu 06:09:03:982.426 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5e50 2026-06-04 Thu 06:09:03:982.461 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5290 2026-06-04 Thu 06:09:03:982.493 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd4770 2026-06-04 Thu 06:09:03:982.547 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5e50 2026-06-04 Thu 06:09:03:982.564 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd5290 2026-06-04 Thu 06:09:03:982.578 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd4770 2026-06-04 Thu 06:09:04:045.780 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd9f90 2026-06-04 Thu 06:09:04:045.956 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd94b0 2026-06-04 Thu 06:09:04:046.004 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbdaa70 2026-06-04 Thu 06:09:04:046.036 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7ef0 2026-06-04 Thu 06:09:04:046.351 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd9f90 2026-06-04 Thu 06:09:04:046.386 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd94b0 2026-06-04 Thu 06:09:04:046.404 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbdaa70 2026-06-04 Thu 06:09:04:046.420 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd7ef0 2026-06-04 Thu 06:09:04:047.309 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd89d0 2026-06-04 Thu 06:09:04:047.661 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd89d0 2026-06-04 Thu 06:09:04:184.648 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbcd180 2026-06-04 Thu 06:09:04:184.813 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2180 2026-06-04 Thu 06:09:04:184.850 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3130 2026-06-04 Thu 06:09:04:185.156 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbcd180 2026-06-04 Thu 06:09:04:185.191 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2180 2026-06-04 Thu 06:09:04:185.204 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2830 2026-06-04 Thu 06:09:04:185.328 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3130 2026-06-04 Thu 06:09:04:185.495 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd2830 2026-06-04 Thu 06:09:04:185.990 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3c50 2026-06-04 Thu 06:09:04:186.296 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: -1 0xbd3c50 2026-06-04 Thu 06:09:06:095.104 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:09:06:095.206 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: SANDESH: Send FAILED: 1780553346094827 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 98 dequeues = 98 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 135 dequeues = 135 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1780553323047847 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1780553323048029 ] max_sm_queue_count = 6 ] 2026-06-04 Thu 06:09:06:095.234 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:09:06:095.244 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:09:06:095.338 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: SANDESH: Send FAILED: 1780553346094962 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 71, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-06-04 Thu 06:09:06:095.353 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568117188160, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:09:08:982.448 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7410 2026-06-04 Thu 06:09:08:982.602 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd7410 2026-06-04 Thu 06:09:08:982.612 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:09:08:982.631 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:09:08:982.640 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd6930 2026-06-04 Thu 06:09:08:982.684 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5e50 2026-06-04 Thu 06:09:08:982.720 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd5290 2026-06-04 Thu 06:09:08:982.757 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd4770 2026-06-04 Thu 06:09:08:982.820 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd6930 2026-06-04 Thu 06:09:08:982.827 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:09:08:982.836 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:09:08:982.843 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd5e50 2026-06-04 Thu 06:09:08:982.847 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:09:08:982.854 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:09:08:982.860 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd5290 2026-06-04 Thu 06:09:08:982.864 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:09:08:982.870 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:09:08:982.877 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd4770 2026-06-04 Thu 06:09:08:982.881 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:09:08:982.887 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:09:08:982.913 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:09:08:982.921 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:09:08:982.988 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:09:08:983.001 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:09:08:983.008 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:09:08:983.087 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:09:08:983.125 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567979705920, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:09:08:983.633 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:09:08:983.700 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:09:08:983.712 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:09:08:983.715 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567979705920, Pid 1]: SANDESH: Send FAILED: 1780553348983036 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 06:09:08:983.719 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:09:08:983.731 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567979705920, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:09:08:983.750 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:09:08:983.757 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:09:08:983.773 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:09:08:983.785 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:09:08:983.802 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:09:09:046.420 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd9f90 2026-06-04 Thu 06:09:09:046.536 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd94b0 2026-06-04 Thu 06:09:09:046.569 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbdaa70 2026-06-04 Thu 06:09:09:046.591 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd7ef0 2026-06-04 Thu 06:09:09:046.857 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd9f90 2026-06-04 Thu 06:09:09:046.865 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:09:09:046.884 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:09:09:046.923 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbdaa70 2026-06-04 Thu 06:09:09:046.928 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:09:09:046.935 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:09:09:046.945 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd7ef0 2026-06-04 Thu 06:09:09:046.950 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:09:09:046.959 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:09:09:046.969 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd94b0 2026-06-04 Thu 06:09:09:046.975 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:09:09:046.984 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:09:09:047.449 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:09:09:047.482 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:09:09:047.497 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:09:09:047.505 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:09:09:047.516 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:09:09:047.538 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:09:09:047.550 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:09:09:047.558 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:09:09:047.579 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:09:09:047.641 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:09:09:047.655 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:09:09:047.667 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:09:09:047.684 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567979705920, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:09:09:047.743 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567979705920, Pid 1]: SANDESH: Send FAILED: 1780553349047606 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 06:09:09:047.753 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567979705920, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:09:09:047.687 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd89d0 2026-06-04 Thu 06:09:09:048.260 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd89d0 2026-06-04 Thu 06:09:09:048.280 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:09:09:048.292 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:09:09:048.538 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:09:09:048.548 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:09:09:048.573 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:09:09:185.231 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbcd180 2026-06-04 Thu 06:09:09:185.349 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2180 2026-06-04 Thu 06:09:09:185.383 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3130 2026-06-04 Thu 06:09:09:185.519 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd2830 2026-06-04 Thu 06:09:09:185.666 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbcd180 2026-06-04 Thu 06:09:09:185.673 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:09:09:185.694 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:09:09:185.751 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd2180 2026-06-04 Thu 06:09:09:185.757 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:09:09:185.765 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:09:09:185.772 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd3130 2026-06-04 Thu 06:09:09:185.776 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:09:09:185.782 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:09:09:185.909 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd2830 2026-06-04 Thu 06:09:09:185.916 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:09:09:185.924 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:09:09:186.321 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xbd3c50 2026-06-04 Thu 06:09:09:186.459 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:09:09:186.469 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:09:09:186.479 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:09:09:186.486 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:09:09:186.506 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:09:09:186.513 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:09:09:186.523 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:09:09:186.530 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:09:09:186.551 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:09:09:186.648 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:09:09:186.660 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:09:09:186.667 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:09:09:186.724 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SANDESH: Send FAILED: 1780553349186575 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 06:09:09:186.733 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:09:09:186.669 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:09:09:186.914 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: RAC_Connect status: 0 0xbd3c50 2026-06-04 Thu 06:09:09:186.918 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:09:09:186.926 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-06-04 Thu 06:09:09:187.578 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-06-04 Thu 06:09:09:187.602 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-06-04 Thu 06:09:09:187.616 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-06-04 Thu 06:09:13:047.982 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568017479232, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-06-04 Thu 06:09:13:048.272 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568017479232, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-06-04 Thu 06:09:13:048.383 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568017479232, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:09:13:048.413 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568017479232, Pid 1]: SANDESH: Send FAILED: 1780553353048087 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 06:09:13:048.422 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568017479232, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:09:13:048.431 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568017479232, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:09:13:048.573 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568017479232, Pid 1]: SANDESH: Send FAILED: 1780553353048365 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 76, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-06-04 Thu 06:09:13:048.600 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568017479232, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:09:13:048.638 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-06-04 Thu 06:09:13:048.669 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: -(-1):EvTcpConnectFail 2026-06-04 Thu 06:09:13:048.746 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-06-04 Thu 06:09:13:048.773 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: Connect : EvTcpConnectFail 2026-06-04 Thu 06:09:13:048.889 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2026-06-04 Thu 06:09:13:048.920 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: Idle 2026-06-04 Thu 06:09:13:048.986 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:09:13:049.101 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: SANDESH: Send FAILED: 1780553353048844 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 78, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-06-04 Thu 06:09:13:049.127 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:09:13:049.137 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:09:13:049.153 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: SANDESH: Send FAILED: 1780553353048870 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 06:09:13:049.159 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:09:13:049.165 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-04 Thu 06:09:13:049.191 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:09:13:049.280 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: SANDESH: Send FAILED: 1780553353048971 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 153, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 78, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-06-04 Thu 06:09:13:049.294 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:09:36:095.752 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-06-04 Thu 06:09:36:095.819 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:09:36:095.867 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: SANDESH: Send FAILED: 1780553376095643 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 108 dequeues = 108 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 148 dequeues = 148 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1780553353048920 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1780553353049190 ] max_sm_queue_count = 6 ] 2026-06-04 Thu 06:09:36:095.878 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:09:36:095.885 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:09:36:095.982 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: SANDESH: Send FAILED: 1780553376095733 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 46 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 153, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 81, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-06-04 Thu 06:09:36:096.003 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568125584960, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:09:41:177.606 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:09:41:177.683 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: SANDESH: Send FAILED: 1780553381177559 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2026-06-04 Thu 06:09:41:177.695 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2026-06-04 Thu 06:09:43:049.423 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-06-04 Thu 06:09:43:049.664 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-06-04 Thu 06:09:43:049.751 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:09:43:049.772 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: SANDESH: Send FAILED: 1780553383049493 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 06:09:43:049.790 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:09:43:049.798 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:09:43:051.643 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: SANDESH: Send FAILED: 1780553383049734 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 153, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 84, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-06-04 Thu 06:09:43:051.663 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:09:43:053.075 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:38878::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 281 2026-06-04 Thu 06:09:43:053.091 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:38878::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2026-06-04 Thu 06:09:43:053.097 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568132727296, Pid 1]: 10.0.0.50:38878::10.0.0.38:8086(-1):EvTcpConnected 2026-06-04 Thu 06:09:43:055.372 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-06-04 Thu 06:09:43:055.393 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-06-04 Thu 06:09:43:055.440 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: ClientInit 2026-06-04 Thu 06:09:43:055.450 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5888-2.:contrail-query-engine:0:Database count 7 2026-06-04 Thu 06:09:43:055.530 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:09:43:055.538 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-04 Thu 06:09:43:055.556 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: SANDESH: Send FAILED: 1780553383055433 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 06:09:43:055.563 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-04 Thu 06:09:43:055.569 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:09:43:055.574 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-06-04 Thu 06:09:43:055.585 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:09:43:055.590 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-04 Thu 06:09:43:055.679 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: SANDESH: Send FAILED: 1780553383055519 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 7 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 49 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 154, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 86, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-06-04 Thu 06:09:43:055.695 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568013280832, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-04 Thu 06:09:43:081.404 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: OnMessage control in state: ClientInit 2026-06-04 Thu 06:09:43:081.457 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-06-04 Thu 06:09:43:081.466 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-06-04 Thu 06:09:43:081.487 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Received Ctrl Message with size 0 2026-06-04 Thu 06:09:43:081.672 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 7 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 161935 calls = 25 average_bytes = 6477 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 108 dequeues = 108 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 148 dequeues = 148 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1780553353048920 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1780553353049190 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 16 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 119853 calls = 14 average_bytes = 8560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 49 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 154, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 86, _iter59->first = sent _iter59->second = 27, ] ] max_sm_queue_count = 6 proxy seq 60 2026-06-04 Thu 06:09:43:081.735 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-06-04 Thu 06:09:43:081.757 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 43 2026-06-04 Thu 06:09:43:081.769 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-06-04 Thu 06:09:43:081.800 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5888-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 5 2026-06-04 Thu 06:09:43:081.813 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2026-06-04 Thu 06:09:43:081.823 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-06-04 Thu 06:09:43:086.406 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: Established 2026-06-04 Thu 06:09:46:982.886 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: OnMessage control in state: Established 2026-06-04 Thu 06:09:46:982.971 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2026-06-04 Thu 06:09:46:982.980 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Established : EvSandeshMessageRecv 2026-06-04 Thu 06:09:46:983.007 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: Received Ctrl Message with size 3 2026-06-04 Thu 06:09:46:983.020 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 61, total = 0 2026-06-04 Thu 06:09:46:983.029 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 45, total = 0 2026-06-04 Thu 06:09:46:983.039 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 5, total = 0 2026-06-04 Thu 06:09:46:983.045 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568025876032, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-06-04 Thu 06:10:36:096.430 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568112989760, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-06-04 Thu 06:11:36:097.362 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568000689728, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2026-06-04 Thu 06:12:36:098.964 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140568112989760, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2026-06-04 Thu 06:13:18:520.793 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: Processing scm::EvStop in state Established 2026-06-04 Thu 06:13:18:520.899 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: Idle 2026-06-04 Thu 06:13:18:520.992 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-06-04 Thu 06:13:18:521.002 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-06-04 Thu 06:13:18:521.118 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: SANDESH: Send FAILED: 1780553598520978 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 7 pid = 1 http_port = 8091 start_time = 1780553136092456 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 384327 calls = 46 average_bytes = 8354 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 26 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 50 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 154, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 88, _iter59->first = sent _iter59->second = 51, ] ] ] 2026-06-04 Thu 06:13:18:521.142 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 140567031838272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:13:44:498.505 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: http-server-port 8091 2026-06-04 Thu 06:13:44:498.572 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Max-tasks 16 2026-06-04 Thu 06:13:44:498.579 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Max-slice 100 2026-06-04 Thu 06:13:44:498.584 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Collectors 10.0.0.38:8086 2026-06-04 Thu 06:13:44:498.588 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Collectors 10.0.0.50:8086 2026-06-04 Thu 06:13:44:498.592 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Collectors 10.0.0.33:8086 2026-06-04 Thu 06:13:44:498.611 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: SANDESH: ROLE : Generator 2026-06-04 Thu 06:13:44:498.616 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-06-04 Thu 06:13:44:498.621 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5888-2. 2026-06-04 Thu 06:13:44:498.625 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: SANDESH: NODE TYPE : Database 2026-06-04 Thu 06:13:44:498.629 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: SANDESH: INSTANCE ID : 0 2026-06-04 Thu 06:13:44:498.633 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-06-04 Thu 06:13:44:498.638 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-06-04 Thu 06:13:44:501.446 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: HTTP Introspect Init 2026-06-04 Thu 06:13:44:501.553 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 138 2026-06-04 Thu 06:13:44:501.628 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Sandesh Http Server Port: 8091 2026-06-04 Thu 06:13:44:501.683 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-06-04 Thu 06:13:44:501.697 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-06-04 Thu 06:13:44:502.104 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Idle 2026-06-04 Thu 06:13:44:502.243 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: SANDESH: No Client: 1780553624502190 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780553624502119 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 06:13:44:502.366 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-06-04 Thu 06:13:44:502.379 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-06-04 Thu 06:13:44:502.389 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Cassandra Servers: 10.0.0.38:9042 10.0.0.50:9042 10.0.0.33:9042 2026-06-04 Thu 06:13:44:502.507 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: Processing scm::EvStart in state Idle 2026-06-04 Thu 06:13:44:502.590 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: Disconnect 2026-06-04 Thu 06:13:44:502.665 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-06-04 Thu 06:13:44:502.920 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-06-04 Thu 06:13:44:502.984 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:13:44:503.001 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: SANDESH: Send FAILED: 1780553624502583 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-06-04 Thu 06:13:44:503.015 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:13:44:503.023 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:13:44:503.091 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: SANDESH: Send FAILED: 1780553624502651 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780553624502119 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 06:13:44:503.101 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:13:44:503.109 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:13:44:503.153 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: SANDESH: Send FAILED: 1780553624502706 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780553624502119 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 06:13:44:503.164 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:13:44:503.171 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:13:44:503.182 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: SANDESH: Send FAILED: 1780553624502740 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-06-04 Thu 06:13:44:503.187 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:13:44:503.196 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:13:44:503.238 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: SANDESH: Send FAILED: 1780553624502970 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1780553624502119 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 06:13:44:503.250 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-06-04 Thu 06:13:44:609.738 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-06-04 Thu 06:13:44:619.105 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791417722432, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-06-04 Thu 06:13:44:619.166 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791417722432, Pid 1]: SANDESH: Send FAILED: 1780553624618024 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-06-04 Thu 06:13:44:619.184 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791417722432, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-06-04 Thu 06:13:44:619.625 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:43442::10.0.0.50:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2026-06-04 Thu 06:13:44:619.659 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: 10.0.0.50:43442::10.0.0.50:8086(-1) OnSessionEvent TCP Connected 2026-06-04 Thu 06:13:44:619.669 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: 10.0.0.50:43442::10.0.0.50:8086(-1):EvTcpConnected 2026-06-04 Thu 06:13:44:619.695 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-06-04 Thu 06:13:44:619.716 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-06-04 Thu 06:13:44:619.764 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: ClientInit 2026-06-04 Thu 06:13:44:619.782 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5888-2.:contrail-query-engine:0:Database count 1 2026-06-04 Thu 06:13:44:619.699 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: RAC_Connect status: 0 0x1ed2430 2026-06-04 Thu 06:13:44:619.834 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:13:44:619.846 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-04 Thu 06:13:44:619.860 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: RAC_Connect status: 0 0x1ed2ff0 2026-06-04 Thu 06:13:44:619.871 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:13:44:619.877 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-04 Thu 06:13:44:619.886 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: RAC_Connect status: 0 0x1ed3c30 2026-06-04 Thu 06:13:44:619.890 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:13:44:619.896 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-04 Thu 06:13:44:619.905 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: RAC_Connect status: 0 0x1ed4870 2026-06-04 Thu 06:13:44:619.909 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:13:44:619.915 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-04 Thu 06:13:44:619.923 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: RAC_Connect status: 0 0x1ed54b0 2026-06-04 Thu 06:13:44:619.934 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:13:44:619.940 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-04 Thu 06:13:44:619.949 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: RAC_Connect status: 0 0x1eca540 2026-06-04 Thu 06:13:44:619.953 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:13:44:619.959 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-04 Thu 06:13:44:619.967 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: RAC_Connect status: 0 0x1ecfdf0 2026-06-04 Thu 06:13:44:619.971 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:13:44:619.977 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-04 Thu 06:13:44:619.986 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: RAC_Connect status: 0 0x1ecf540 2026-06-04 Thu 06:13:44:619.990 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:13:44:619.996 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-04 Thu 06:13:44:620.004 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: RAC_Connect status: 0 0x1ed0ab0 2026-06-04 Thu 06:13:44:620.008 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:13:44:620.014 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-04 Thu 06:13:44:620.022 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: RAC_Connect status: 0 0x1ed1770 2026-06-04 Thu 06:13:44:620.025 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:13:44:620.031 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-04 Thu 06:13:44:620.039 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: RAC_Connect status: 0 0x1ed60f0 2026-06-04 Thu 06:13:44:620.043 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:13:44:620.049 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-04 Thu 06:13:44:620.057 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: RAC_Connect status: 0 0x1ed6d30 2026-06-04 Thu 06:13:44:620.061 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:13:44:620.067 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-04 Thu 06:13:44:620.093 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: RAC_Connect status: 0 0x1ed7970 2026-06-04 Thu 06:13:44:620.098 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:13:44:620.104 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-04 Thu 06:13:44:620.112 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: RAC_Connect status: 0 0x1ed85b0 2026-06-04 Thu 06:13:44:620.116 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:13:44:620.122 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-04 Thu 06:13:44:620.130 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: RAC_Connect status: 0 0x1ed91f0 2026-06-04 Thu 06:13:44:620.134 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: Connected to REDIS... 2026-06-04 Thu 06:13:44:620.140 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-06-04 Thu 06:13:44:620.362 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-04 Thu 06:13:44:620.374 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-04 Thu 06:13:44:620.411 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-04 Thu 06:13:44:620.425 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-04 Thu 06:13:44:620.433 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-04 Thu 06:13:44:619.863 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:13:44:620.464 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-04 Thu 06:13:44:620.481 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: SANDESH: Send FAILED: 1780553624619756 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-06-04 Thu 06:13:44:620.489 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-04 Thu 06:13:44:620.502 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:13:44:620.508 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-06-04 Thu 06:13:44:620.527 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:13:44:620.533 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-06-04 Thu 06:13:44:620.613 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: SANDESH: Send FAILED: 1780553624619850 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780553624502119 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 3, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-06-04 Thu 06:13:44:620.645 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-06-04 Thu 06:13:44:620.668 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:13:44:620.674 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-04 Thu 06:13:44:620.714 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: SANDESH: Send FAILED: 1780553624620403 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-06-04 Thu 06:13:44:620.721 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-04 Thu 06:13:44:621.247 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-04 Thu 06:13:44:621.301 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-04 Thu 06:13:44:621.309 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-04 Thu 06:13:44:621.361 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790489990720, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:13:44:621.399 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790489990720, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-04 Thu 06:13:44:621.430 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790489990720, Pid 1]: SANDESH: Send FAILED: 1780553624621230 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 06:13:44:621.447 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790489990720, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-04 Thu 06:13:44:621.752 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-04 Thu 06:13:44:621.762 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-04 Thu 06:13:44:621.773 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-04 Thu 06:13:44:621.780 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-04 Thu 06:13:44:621.866 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-04 Thu 06:13:44:621.881 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-04 Thu 06:13:44:621.891 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-04 Thu 06:13:44:621.910 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-04 Thu 06:13:44:621.917 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-04 Thu 06:13:44:621.927 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-04 Thu 06:13:44:621.933 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-04 Thu 06:13:44:621.943 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-04 Thu 06:13:44:621.950 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-04 Thu 06:13:44:621.966 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-04 Thu 06:13:44:621.973 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-04 Thu 06:13:44:621.982 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-04 Thu 06:13:44:621.989 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-04 Thu 06:13:44:621.999 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-04 Thu 06:13:44:622.006 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-04 Thu 06:13:44:622.015 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-04 Thu 06:13:44:622.022 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-04 Thu 06:13:44:622.031 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-04 Thu 06:13:44:622.039 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-04 Thu 06:13:44:622.049 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-04 Thu 06:13:44:622.055 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-04 Thu 06:13:44:622.064 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-06-04 Thu 06:13:44:622.093 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-06-04 Thu 06:13:44:622.110 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-04 Thu 06:13:44:622.143 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-04 Thu 06:13:44:622.154 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-04 Thu 06:13:44:622.167 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-04 Thu 06:13:44:622.176 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-04 Thu 06:13:44:622.190 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-04 Thu 06:13:44:622.205 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-04 Thu 06:13:44:622.215 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-04 Thu 06:13:44:622.223 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-04 Thu 06:13:44:622.232 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-06-04 Thu 06:13:44:622.284 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:13:44:622.292 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-04 Thu 06:13:44:622.306 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: SANDESH: Send FAILED: 1780553624622130 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 06:13:44:622.312 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-04 Thu 06:13:44:622.319 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-06-04 Thu 06:13:44:622.324 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-06-04 Thu 06:13:44:622.335 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: SANDESH: Send FAILED: 1780553624622201 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-06-04 Thu 06:13:44:622.344 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-06-04 Thu 06:13:44:625.738 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791454000704, Pid 1]: OnMessage control in state: ClientInit 2026-06-04 Thu 06:13:44:625.782 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791454000704, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-06-04 Thu 06:13:44:625.790 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791454000704, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-06-04 Thu 06:13:44:625.805 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791454000704, Pid 1]: Received Ctrl Message with size 0 2026-06-04 Thu 06:13:44:625.947 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791454000704, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5888-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1780553624502119 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 3, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2026-06-04 Thu 06:13:44:625.977 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791454000704, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-06-04 Thu 06:13:44:625.996 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791454000704, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5888-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2026-06-04 Thu 06:13:44:626.006 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791454000704, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-06-04 Thu 06:13:44:626.012 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791454000704, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-06-04 Thu 06:13:44:626.018 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791454000704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-06-04 Thu 06:13:44:626.053 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791454000704, Pid 1]: Established 2026-06-04 Thu 06:14:44:502.851 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791392261696, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-06-04 Thu 06:15:44:503.702 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791396730432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-06-04 Thu 06:16:44:504.310 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791417722432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-06-04 Thu 06:17:44:504.976 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791413524032, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-06-04 Thu 06:18:44:505.686 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-06-04 Thu 06:19:44:506.662 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790489990720, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2026-06-04 Thu 06:20:44:507.604 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790498387520, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2026-06-04 Thu 06:21:44:508.266 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791454000704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2026-06-04 Thu 06:22:44:508.881 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790494189120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2026-06-04 Thu 06:23:44:509.583 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791454000704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2026-06-04 Thu 06:24:44:510.317 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2026-06-04 Thu 06:25:44:511.081 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790489990720, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2026-06-04 Thu 06:26:44:511.755 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2026-06-04 Thu 06:27:44:512.421 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791445603904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2026-06-04 Thu 06:28:44:513.086 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791417722432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2026-06-04 Thu 06:29:44:513.664 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790498387520, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2026-06-04 Thu 06:29:57:393.350 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.50:52234 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-06-04 Thu 06:30:03:286.787 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.50:46680 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-06-04 Thu 06:30:44:514.348 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790498387520, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2026-06-04 Thu 06:31:27:097.287 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.38:50874 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-06-04 Thu 06:31:44:515.030 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791396730432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2026-06-04 Thu 06:31:50:772.594 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.38 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 812 2026-06-04 Thu 06:31:50:777.642 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID 11e428aa-5fdf-11f1-8602-00000a000026 src/contrail-analytics/contrail-query-engine/query.cc 1165 2026-06-04 Thu 06:31:50:777.767 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QEQueryLog: 11e428aa-5fdf-11f1-8602-00000a000026 :0:json query is: { end_time: 1780554710704758 , enqueue_time: 1780554710773188 , query_metadata: {"enqueue_time": 1780554710773067} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1780554691581498 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "a1a7de84-ba54-4413-8fc4-daf6792cce3d", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-06-04 Thu 06:31:50:777.987 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 936 2026-06-04 Thu 06:31:50:778.023 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 986 2026-06-04 Thu 06:31:50:778.084 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for 11e428aa-5fdf-11f1-8602-00000a000026 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 994 2026-06-04 Thu 06:31:50:778.254 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: [SYS_INFO]: QEQueryLog: 11e428aa-5fdf-11f1-8602-00000a000026 :0:json query is: { end_time: 1780554710704758 , enqueue_time: 1780554710773188 , query_metadata: {"enqueue_time": 1780554710773067} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1780554691581498 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "a1a7de84-ba54-4413-8fc4-daf6792cce3d", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-06-04 Thu 06:31:50:839.124 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:50:839.185 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:50:839.193 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:50:839.243 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:50:839.251 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:50:839.257 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:50:839.263 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:50:839.272 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:50:839.278 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:50:839.285 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:50:839.678 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:50:839.690 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:50:839.697 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:50:839.703 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:50:839.716 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:50:839.723 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:50:839.729 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:50:839.736 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:50:839.742 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 UNION between tables of sizes 0 and 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:50:839.763 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:50:840.028 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QEQueryLog: 11e428aa-5fdf-11f1-8602-00000a000026 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id a1a7de84-ba54-4413-8fc4-daf6792cce3d src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-06-04 Thu 06:31:50:840.125 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QEQueryLog: 11e428aa-5fdf-11f1-8602-00000a000026 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id a1a7de84-ba54-4413-8fc4-daf6792cce3d src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-06-04 Thu 06:31:50:840.221 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QEQueryLog: 11e428aa-5fdf-11f1-8602-00000a000026 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id a1a7de84-ba54-4413-8fc4-daf6792cce3d src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-06-04 Thu 06:31:50:840.302 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QEQueryLog: 11e428aa-5fdf-11f1-8602-00000a000026 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id a1a7de84-ba54-4413-8fc4-daf6792cce3d src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-06-04 Thu 06:31:50:840.386 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QEQueryLog: 11e428aa-5fdf-11f1-8602-00000a000026 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id a1a7de84-ba54-4413-8fc4-daf6792cce3d src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-06-04 Thu 06:31:50:841.881 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791433008704, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:50:841.911 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791433008704, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:50:841.918 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791433008704, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:50:841.924 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791433008704, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:50:841.930 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791433008704, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:50:841.945 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791433008704, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:50:841.951 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791433008704, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:50:841.965 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791433008704, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:50:841.971 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791433008704, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:50:841.977 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791433008704, Pid 1]: [SYS_INFO]: QELog: 11e428aa-5fdf-11f1-8602-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:50:842.038 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791433008704, Pid 1]: [SYS_INFO]: QEQueryLog: 11e428aa-5fdf-11f1-8602-00000a000026 :0:json query is: { end_time: 1780554710704758 , enqueue_time: 1780554710773188 , query_metadata: {"enqueue_time": 1780554710773067} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1780554691581498 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "a1a7de84-ba54-4413-8fc4-daf6792cce3d", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-06-04 Thu 06:31:50:842.334 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791417722432, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 542 2026-06-04 Thu 06:31:50:842.476 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791417722432, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 551 2026-06-04 Thu 06:31:50:842.586 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791417722432, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":5} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 586 2026-06-04 Thu 06:31:50:844.444 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: Finished: QID 11e428aa-5fdf-11f1-8602-00000a000026 Table ObjectVMTable Time(ms) 66 RedisTime(ms) 0 MergeTime(ms) 0 Rows 5 EnQ-delay4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 661 2026-06-04 Thu 06:31:50:844.505 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790494189120, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-5888-2. table = ObjectVMTable query_stats= [ qid = 11e428aa-5fdf-11f1-8602-00000a000026 where = [[{"name": "ObjectId", "value": "a1a7de84-ba54-4413-8fc4-daf6792cce3d", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 63,0, 60,0, 58,0, chunk_select_time = 0,0, 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 66 rows = 5 enq_delay = 4 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 657 2026-06-04 Thu 06:31:50:844.534 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 699 2026-06-04 Thu 06:31:53:862.715 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.38 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 812 2026-06-04 Thu 06:31:53:863.425 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID 13bb7d94-5fdf-11f1-ac27-00000a000026 src/contrail-analytics/contrail-query-engine/query.cc 1165 2026-06-04 Thu 06:31:53:863.477 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QEQueryLog: 13bb7d94-5fdf-11f1-ac27-00000a000026 :0:json query is: { end_time: 1780554713848403 , enqueue_time: 1780554713862885 , query_metadata: {"enqueue_time": 1780554713862610} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1780554691581498 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-98789151:ctest-vn22-29431496:ctest-vn22-29431496", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-06-04 Thu 06:31:53:863.610 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 936 2026-06-04 Thu 06:31:53:863.624 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 986 2026-06-04 Thu 06:31:53:863.662 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for 13bb7d94-5fdf-11f1-ac27-00000a000026 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 994 2026-06-04 Thu 06:31:53:863.756 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QEQueryLog: 13bb7d94-5fdf-11f1-ac27-00000a000026 :0:json query is: { end_time: 1780554713848403 , enqueue_time: 1780554713862885 , query_metadata: {"enqueue_time": 1780554713862610} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1780554691581498 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-98789151:ctest-vn22-29431496:ctest-vn22-29431496", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-06-04 Thu 06:31:53:994.162 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:53:994.222 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:53:994.234 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:53:994.243 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:53:994.252 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:53:994.260 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:53:994.268 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:53:994.277 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:53:994.285 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:53:994.293 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:53:994.428 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QEQueryLog: 13bb7d94-5fdf-11f1-ac27-00000a000026 :0:json query is: { end_time: 1780554713848403 , enqueue_time: 1780554713862885 , query_metadata: {"enqueue_time": 1780554713862610} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1780554691581498 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-98789151:ctest-vn22-29431496:ctest-vn22-29431496", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-06-04 Thu 06:31:53:997.482 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790489990720, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:53:997.514 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790489990720, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:53:997.534 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790489990720, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:53:997.542 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790489990720, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:53:997.551 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790489990720, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:53:997.559 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790489990720, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:53:997.568 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790489990720, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:53:997.576 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790489990720, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:53:997.584 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790489990720, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:53:997.592 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790489990720, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:53:999.448 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:53:999.477 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:53:999.487 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:53:999.497 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:53:999.506 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:53:999.515 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:53:999.525 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:53:999.541 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:53:999.550 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-06-04 Thu 06:31:53:999.559 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: 13bb7d94-5fdf-11f1-ac27-00000a000026 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-06-04 Thu 06:31:53:999.831 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QEQueryLog: 13bb7d94-5fdf-11f1-ac27-00000a000026 ObjectRoutingInstance :1:Column ObjectTypeName1 Table ObjectRoutingInstance object_id default-domain:ctest-AnalyticsBasicTestSanity-98789151:ctest-vn22-29431496:ctest-vn22-29431496 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-06-04 Thu 06:31:54:000.068 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 542 2026-06-04 Thu 06:31:54:000.119 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 551 2026-06-04 Thu 06:31:54:001.013 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":1} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 586 2026-06-04 Thu 06:31:54:001.358 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791454000704, Pid 1]: [SYS_INFO]: QELog: Finished: QID 13bb7d94-5fdf-11f1-ac27-00000a000026 Table ObjectRoutingInstance Time(ms) 137 RedisTime(ms) 0 MergeTime(ms) 0 Rows 1 EnQ-delay0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 661 2026-06-04 Thu 06:31:54:001.440 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791454000704, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 699 2026-06-04 Thu 06:31:54:001.491 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791454000704, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-5888-2. table = ObjectRoutingInstance query_stats= [ qid = 13bb7d94-5fdf-11f1-ac27-00000a000026 where = [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-98789151:ctest-vn22-29431496:ctest-vn22-29431496", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 130,0, 133,0, 135,0, chunk_select_time = 0,0, 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 137 rows = 1 enq_delay = 0 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 657 2026-06-04 Thu 06:32:44:515.695 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791413524032, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2026-06-04 Thu 06:33:44:516.198 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791400928832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2026-06-04 Thu 06:34:44:516.962 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2026-06-04 Thu 06:35:44:517.697 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790489990720, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2026-06-04 Thu 06:36:44:518.301 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791454000704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2026-06-04 Thu 06:37:44:519.067 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2026-06-04 Thu 06:38:44:519.746 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2026-06-04 Thu 06:39:44:520.546 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790489990720, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2026-06-04 Thu 06:40:44:521.280 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791454000704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2026-06-04 Thu 06:41:44:522.102 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791417722432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2026-06-04 Thu 06:42:44:522.679 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139790498387520, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0 2026-06-04 Thu 06:43:41:526.959 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.38:40060 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-06-04 Thu 06:43:44:523.301 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 30 without seqno , total = 0 2026-06-04 Thu 06:44:44:524.103 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791449802304, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 31 without seqno , total = 0 2026-06-04 Thu 06:45:44:524.808 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791441405504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 32 without seqno , total = 0 2026-06-04 Thu 06:45:45:929.438 UTC cn-jenkins-deploy-platform-ansible-os-5888-2 [Thread 139791461140608, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.50:56996 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256