2025-11-27 Thu 01:37:34:833.645 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: http-server-port 8091 2025-11-27 Thu 01:37:34:833.702 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: Max-tasks 16 2025-11-27 Thu 01:37:34:833.709 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: Max-slice 100 2025-11-27 Thu 01:37:34:833.714 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: Collectors 10.0.0.129:8086 2025-11-27 Thu 01:37:34:833.735 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: SANDESH: ROLE : Generator 2025-11-27 Thu 01:37:34:833.740 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-11-27 Thu 01:37:34:833.744 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-juju-k8s-1739-1 2025-11-27 Thu 01:37:34:833.748 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: SANDESH: NODE TYPE : Database 2025-11-27 Thu 01:37:34:833.752 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: SANDESH: INSTANCE ID : 0 2025-11-27 Thu 01:37:34:833.757 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-11-27 Thu 01:37:34:833.761 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-11-27 Thu 01:37:34:837.016 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: HTTP Introspect Init 2025-11-27 Thu 01:37:34:837.462 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-11-27 Thu 01:37:34:839.018 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: Sandesh Http Server Port: 8091 2025-11-27 Thu 01:37:34:840.284 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-11-27 Thu 01:37:34:840.303 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-11-27 Thu 01:37:34:841.834 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: Idle 2025-11-27 Thu 01:37:34:841.970 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: SANDESH: No Client: 1764207454841915 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-11-27 Thu 01:37:34:842.122 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-11-27 Thu 01:37:34:842.135 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-11-27 Thu 01:37:34:842.144 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: Cassandra Servers: 10.0.0.129:9042 2025-11-27 Thu 01:37:34:845.330 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-11-27 Thu 01:37:34:845.351 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:37:34:845.384 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Processing scm::EvStart in state Idle 2025-11-27 Thu 01:37:34:845.410 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:37:34:847.202 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Disconnect 2025-11-27 Thu 01:37:34:847.280 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-11-27 Thu 01:37:34:847.448 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Connect : Start Connect timer 10.0.0.129:8086 2025-11-27 Thu 01:37:34:847.498 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-11-27 Thu 01:37:34:847.516 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: SANDESH: Send FAILED: 1764207454845398 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Redis-Query:Query,Collector ], ] ] ] 2025-11-27 Thu 01:37:34:847.526 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-11-27 Thu 01:37:34:847.533 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-11-27 Thu 01:37:34:847.547 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: SANDESH: Send FAILED: 1764207454845433 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-11-27 Thu 01:37:34:847.557 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-11-27 Thu 01:37:34:847.564 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-11-27 Thu 01:37:34:847.627 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: SANDESH: Send FAILED: 1764207454847269 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 4, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-11-27 Thu 01:37:34:847.638 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-11-27 Thu 01:37:34:847.646 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-11-27 Thu 01:37:34:847.695 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: SANDESH: Send FAILED: 1764207454847311 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 4, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-11-27 Thu 01:37:34:847.703 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-11-27 Thu 01:37:34:847.709 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-11-27 Thu 01:37:34:847.724 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: SANDESH: Send FAILED: 1764207454847333 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-11-27 Thu 01:37:34:847.729 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-11-27 Thu 01:37:34:847.735 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-11-27 Thu 01:37:34:847.778 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: SANDESH: Send FAILED: 1764207454847490 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 4, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-11-27 Thu 01:37:34:847.787 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-11-27 Thu 01:37:39:847.083 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-11-27 Thu 01:37:39:847.138 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:37:39:847.153 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:37:44:848.848 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-11-27 Thu 01:37:44:848.882 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:37:44:848.898 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:37:50:132.117 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:37:50:132.177 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:37:50:132.200 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:37:55:222.590 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:37:55:222.636 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:37:55:222.656 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:38:00:248.317 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:38:00:248.361 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:38:00:248.387 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:38:05:275.003 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:38:05:275.048 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:38:05:275.063 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:38:10:299.125 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:38:10:299.175 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:38:10:299.191 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 7 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:38:15:326.596 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:38:15:326.647 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:38:15:326.662 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 8 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:38:20:351.835 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:38:20:351.872 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:38:20:351.896 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 9 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:38:25:383.512 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:38:25:383.563 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:38:25:383.582 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 10 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:38:30:406.467 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:38:30:406.519 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:38:30:406.536 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 11 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:38:35:426.064 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:38:35:426.114 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:38:35:426.128 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 12 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:38:40:446.853 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:38:40:446.902 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:38:40:446.920 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 13 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:38:45:473.034 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:38:45:473.078 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:38:45:473.093 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 14 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:38:50:505.515 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:38:50:505.559 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:38:50:505.574 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 15 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:38:55:532.610 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:38:55:532.665 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:38:55:532.686 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 16 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:39:00:552.542 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:39:00:552.589 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:39:00:552.604 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 17 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:39:05:574.060 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:39:05:574.113 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:39:05:574.128 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 18 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:39:10:595.227 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:39:10:595.275 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:39:10:595.289 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 19 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:39:15:614.881 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:39:15:614.931 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:39:15:614.946 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 20 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:39:20:631.766 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:39:20:631.816 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:39:20:631.831 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 21 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:39:25:654.027 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:39:25:654.065 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:39:25:654.100 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 22 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:39:30:674.959 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:39:30:674.995 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:39:30:675.016 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 23 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:39:35:710.453 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:39:35:710.511 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:39:35:710.526 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 24 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:39:40:729.183 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:39:40:729.223 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:39:40:729.237 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 25 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:39:45:748.777 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:39:45:748.815 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:39:45:748.829 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 26 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:39:50:766.597 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:39:50:766.636 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:39:50:766.661 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 27 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:39:55:796.714 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:39:55:796.756 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:39:55:796.772 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 28 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:40:00:815.895 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:40:00:815.943 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:40:00:815.958 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 29 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:40:05:841.519 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:40:05:841.557 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:40:05:841.572 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 30 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:40:10:862.142 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:40:10:862.199 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:40:10:862.213 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 31 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:40:15:880.176 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:40:15:880.210 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:40:15:880.224 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 32 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:40:20:902.498 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:40:20:902.546 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:40:20:902.571 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 33 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:40:25:923.257 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:40:25:923.299 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:40:25:923.318 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 34 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:40:30:954.014 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:40:30:954.058 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:40:30:954.073 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 35 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:40:35:971.523 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:40:35:971.568 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:40:35:971.583 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 36 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:40:40:993.336 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:40:40:993.392 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:40:40:993.406 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 37 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:40:46:009.453 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:40:46:009.522 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:40:46:009.537 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 38 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:40:51:025.674 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:40:51:025.731 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:40:51:025.745 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 39 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:40:56:079.753 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1003 2025-11-27 Thu 01:40:56:079.795 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:40:56:079.809 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 40 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:41:01:153.181 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-11-27 Thu 01:41:01:163.521 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 41 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:41:06:214.913 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-11-27 Thu 01:41:06:214.958 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 42 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-11-27 Thu 01:41:11:308.865 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-11-27 Thu 01:41:16:311.678 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-11-27 Thu 01:41:16:311.737 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: SANDESH: Send FAILED: 1764207676311553 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-11-27 Thu 01:41:16:311.745 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-11-27 Thu 01:41:16:313.154 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:36992 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:41:16:313.281 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: RAC_Connect status: -1 0x102aaf0 2025-11-27 Thu 01:41:16:313.323 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: RAC_Connect status: -1 0x10dfa70 2025-11-27 Thu 01:41:16:313.339 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: RAC_Connect status: -1 0x10e0c30 2025-11-27 Thu 01:41:16:313.356 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: RAC_Connect status: -1 0x10e1710 2025-11-27 Thu 01:41:16:313.372 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: RAC_Connect status: -1 0x10e2230 2025-11-27 Thu 01:41:16:313.395 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-11-27 Thu 01:41:16:313.400 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: -(-1):EvTcpConnectFail 2025-11-27 Thu 01:41:16:313.444 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-11-27 Thu 01:41:16:313.471 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Connect : EvTcpConnectFail 2025-11-27 Thu 01:41:16:313.545 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-11-27 Thu 01:41:16:313.580 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Idle 2025-11-27 Thu 01:41:16:314.138 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-11-27 Thu 01:41:16:314.182 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: SANDESH: Send FAILED: 1764207676313523 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-11-27 Thu 01:41:16:314.189 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-11-27 Thu 01:41:16:314.196 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-11-27 Thu 01:41:16:314.235 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-11-27 Thu 01:41:16:314.316 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: SANDESH: Send FAILED: 1764207676314120 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 128 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 131, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-11-27 Thu 01:41:16:314.333 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-11-27 Thu 01:41:16:314.437 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-11-27 Thu 01:41:16:314.464 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: SANDESH: Send FAILED: 1764207676314372 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1764207676313579 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1764207676314234 ] max_sm_queue_count = 6 ] 2025-11-27 Thu 01:41:16:314.474 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-11-27 Thu 01:41:16:314.480 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-11-27 Thu 01:41:16:314.545 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: SANDESH: Send FAILED: 1764207676314421 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 128 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 131, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 9, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-11-27 Thu 01:41:16:314.557 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-11-27 Thu 01:41:16:314.844 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:57824 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:41:16:316.209 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:47408 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:41:16:318.599 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:46426 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:41:16:318.751 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:36992 > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-11-27 Thu 01:41:16:321.082 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:60198 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:41:16:322.420 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:57824 > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-11-27 Thu 01:41:16:325.042 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:44180 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:41:16:328.663 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:47780 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:41:16:329.423 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:47408 > Write failed due to error: Broken pipe src/contrail-common/io/tcp_session.cc 384 2025-11-27 Thu 01:41:16:329.442 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:47408 > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-11-27 Thu 01:41:16:329.859 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:46426 > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-11-27 Thu 01:41:16:331.513 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:45358 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:41:16:332.850 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:60198 > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-11-27 Thu 01:41:16:335.385 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:44180 > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-11-27 Thu 01:41:16:335.433 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:47780 > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 313 2025-11-27 Thu 01:41:17:315.730 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-11-27 Thu 01:41:17:315.953 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Connect : Start Connect timer 10.0.0.129:8086 2025-11-27 Thu 01:41:17:316.029 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-11-27 Thu 01:41:17:316.051 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: SANDESH: Send FAILED: 1764207677315797 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-11-27 Thu 01:41:17:316.058 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-11-27 Thu 01:41:17:316.066 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-11-27 Thu 01:41:17:316.145 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: SANDESH: Send FAILED: 1764207677316017 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 128 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 146, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 11, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-11-27 Thu 01:41:17:316.175 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568715175488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-11-27 Thu 01:41:17:319.121 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:39228::10.0.0.129:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 281 2025-11-27 Thu 01:41:17:319.139 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: 10.0.0.129:39228::10.0.0.129:8086(-1) OnSessionEvent TCP Connected 2025-11-27 Thu 01:41:17:319.146 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: 10.0.0.129:39228::10.0.0.129:8086(-1):EvTcpConnected 2025-11-27 Thu 01:41:17:319.379 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-11-27 Thu 01:41:17:320.083 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-11-27 Thu 01:41:17:320.124 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: ClientInit 2025-11-27 Thu 01:41:17:320.141 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-juju-k8s-1739-1:contrail-query-engine:0:Database count 1 2025-11-27 Thu 01:41:17:320.219 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-11-27 Thu 01:41:17:320.228 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-11-27 Thu 01:41:17:320.244 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: SANDESH: Send FAILED: 1764207677320118 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-11-27 Thu 01:41:17:320.250 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-11-27 Thu 01:41:17:320.257 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-11-27 Thu 01:41:17:320.262 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-11-27 Thu 01:41:17:320.280 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-11-27 Thu 01:41:17:320.285 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-11-27 Thu 01:41:17:320.361 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: SANDESH: Send FAILED: 1764207677320209 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 128 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 147, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 13, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-11-27 Thu 01:41:17:320.380 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-11-27 Thu 01:41:17:321.769 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: 10.0.0.129:39228::10.0.0.129:8086(-1) OnSessionEvent TCP Connection Closed 2025-11-27 Thu 01:41:17:321.784 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: 10.0.0.129:39228::10.0.0.129:8086(-1):EvTcpClose 2025-11-27 Thu 01:41:17:321.818 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-11-27 Thu 01:41:17:321.828 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: ClientInit : EvTcpClose 2025-11-27 Thu 01:41:17:321.863 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Return to idle with 1000 2025-11-27 Thu 01:41:17:321.869 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Idle 2025-11-27 Thu 01:41:17:321.933 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-11-27 Thu 01:41:17:321.951 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: SANDESH: Send FAILED: 1764207677321850 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-11-27 Thu 01:41:17:321.958 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-11-27 Thu 01:41:17:321.965 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-11-27 Thu 01:41:17:321.991 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-11-27 Thu 01:41:17:322.072 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: SANDESH: Send FAILED: 1764207677321922 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 1 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 1567 calls = 1 average_bytes = 1567 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 128 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 147, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 15, _iter59->first = sent _iter59->second = 1, ] ] ] 2025-11-27 Thu 01:41:17:322.090 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-11-27 Thu 01:41:18:321.989 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-11-27 Thu 01:41:18:322.235 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: Connect : Start Connect timer 10.0.0.129:8086 2025-11-27 Thu 01:41:18:322.317 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-11-27 Thu 01:41:18:322.337 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: SANDESH: Send FAILED: 1764207678322063 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-11-27 Thu 01:41:18:322.344 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-11-27 Thu 01:41:18:322.351 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-11-27 Thu 01:41:18:322.447 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: SANDESH: Send FAILED: 1764207678322303 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 1567 calls = 1 average_bytes = 1567 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 128 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 147, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 17, _iter59->first = sent _iter59->second = 1, ] ] ] 2025-11-27 Thu 01:41:18:322.464 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568695690816, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-11-27 Thu 01:41:18:325.539 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:39236::10.0.0.129:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 281 2025-11-27 Thu 01:41:18:325.563 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: 10.0.0.129:39236::10.0.0.129:8086(-1) OnSessionEvent TCP Connected 2025-11-27 Thu 01:41:18:325.570 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: 10.0.0.129:39236::10.0.0.129:8086(-1):EvTcpConnected 2025-11-27 Thu 01:41:18:326.923 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-11-27 Thu 01:41:18:326.955 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-11-27 Thu 01:41:18:326.988 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: ClientInit 2025-11-27 Thu 01:41:18:326.999 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-juju-k8s-1739-1:contrail-query-engine:0:Database count 2 2025-11-27 Thu 01:41:18:327.066 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-11-27 Thu 01:41:18:327.074 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-11-27 Thu 01:41:18:327.089 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: SANDESH: Send FAILED: 1764207678326982 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-11-27 Thu 01:41:18:327.095 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-11-27 Thu 01:41:18:327.102 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-11-27 Thu 01:41:18:327.107 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-11-27 Thu 01:41:18:327.116 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-11-27 Thu 01:41:18:327.121 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-11-27 Thu 01:41:18:327.210 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: SANDESH: Send FAILED: 1764207678327054 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 1567 calls = 1 average_bytes = 1567 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 128 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 148, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 19, _iter59->first = sent _iter59->second = 1, ] ] ] 2025-11-27 Thu 01:41:18:327.223 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-11-27 Thu 01:41:18:330.530 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: 10.0.0.129:39236::10.0.0.129:8086(-1) OnSessionEvent TCP Connection Closed 2025-11-27 Thu 01:41:18:330.546 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: 10.0.0.129:39236::10.0.0.129:8086(-1):EvTcpClose 2025-11-27 Thu 01:41:18:331.257 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-11-27 Thu 01:41:18:332.533 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: ClientInit : EvTcpClose 2025-11-27 Thu 01:41:18:332.602 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Return to idle with 1000 2025-11-27 Thu 01:41:18:332.609 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Idle 2025-11-27 Thu 01:41:18:332.680 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-11-27 Thu 01:41:18:332.704 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: SANDESH: Send FAILED: 1764207678332582 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-11-27 Thu 01:41:18:332.714 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-11-27 Thu 01:41:18:332.724 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-11-27 Thu 01:41:18:332.800 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-11-27 Thu 01:41:18:332.883 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: SANDESH: Send FAILED: 1764207678332668 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 2 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3134 calls = 2 average_bytes = 1567 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 128 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 148, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 21, _iter59->first = sent _iter59->second = 2, ] ] ] 2025-11-27 Thu 01:41:18:332.898 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-11-27 Thu 01:41:19:332.735 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-11-27 Thu 01:41:19:332.980 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: Connect : Start Connect timer 10.0.0.129:8086 2025-11-27 Thu 01:41:19:333.074 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-11-27 Thu 01:41:19:333.096 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: SANDESH: Send FAILED: 1764207679332804 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-11-27 Thu 01:41:19:333.109 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-11-27 Thu 01:41:19:333.116 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-11-27 Thu 01:41:19:333.214 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: SANDESH: Send FAILED: 1764207679333059 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3134 calls = 2 average_bytes = 1567 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 128 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 148, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 2, ] ] ] 2025-11-27 Thu 01:41:19:333.231 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-11-27 Thu 01:41:19:335.978 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:39250::10.0.0.129:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 281 2025-11-27 Thu 01:41:19:336.140 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: 10.0.0.129:39250::10.0.0.129:8086(-1) OnSessionEvent TCP Connected 2025-11-27 Thu 01:41:19:336.149 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: 10.0.0.129:39250::10.0.0.129:8086(-1):EvTcpConnected 2025-11-27 Thu 01:41:19:336.213 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567904253504, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-11-27 Thu 01:41:19:336.226 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567904253504, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-11-27 Thu 01:41:19:336.258 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567904253504, Pid 1]: ClientInit 2025-11-27 Thu 01:41:19:336.267 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567904253504, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-juju-k8s-1739-1:contrail-query-engine:0:Database count 3 2025-11-27 Thu 01:41:19:336.339 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567904253504, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-11-27 Thu 01:41:19:336.347 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567904253504, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-11-27 Thu 01:41:19:336.363 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567904253504, Pid 1]: SANDESH: Send FAILED: 1764207679336252 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-11-27 Thu 01:41:19:336.375 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567904253504, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-11-27 Thu 01:41:19:336.382 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567904253504, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-11-27 Thu 01:41:19:336.387 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567904253504, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-11-27 Thu 01:41:19:336.405 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567904253504, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-11-27 Thu 01:41:19:336.410 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567904253504, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-11-27 Thu 01:41:19:336.490 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567904253504, Pid 1]: SANDESH: Send FAILED: 1764207679336329 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3134 calls = 2 average_bytes = 1567 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 128 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 149, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 25, _iter59->first = sent _iter59->second = 2, ] ] ] 2025-11-27 Thu 01:41:19:336.508 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567904253504, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-11-27 Thu 01:41:19:337.599 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: 10.0.0.129:39250::10.0.0.129:8086(-1) OnSessionEvent TCP Connection Closed 2025-11-27 Thu 01:41:19:337.615 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: 10.0.0.129:39250::10.0.0.129:8086(-1):EvTcpClose 2025-11-27 Thu 01:41:19:337.649 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-11-27 Thu 01:41:19:337.656 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: ClientInit : EvTcpClose 2025-11-27 Thu 01:41:19:337.692 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Return to idle with 1000 2025-11-27 Thu 01:41:19:337.698 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Idle 2025-11-27 Thu 01:41:19:337.749 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-11-27 Thu 01:41:19:337.765 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: SANDESH: Send FAILED: 1764207679337679 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-11-27 Thu 01:41:19:337.784 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-11-27 Thu 01:41:19:337.790 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-11-27 Thu 01:41:19:337.816 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-11-27 Thu 01:41:19:337.893 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: SANDESH: Send FAILED: 1764207679337739 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 4701 calls = 3 average_bytes = 1567 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 128 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 149, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 27, _iter59->first = sent _iter59->second = 3, ] ] ] 2025-11-27 Thu 01:41:19:337.905 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-11-27 Thu 01:41:20:338.203 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-11-27 Thu 01:41:20:338.481 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: Connect : Start Connect timer 10.0.0.129:8086 2025-11-27 Thu 01:41:20:338.577 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-11-27 Thu 01:41:20:338.600 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: SANDESH: Send FAILED: 1764207680338273 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-11-27 Thu 01:41:20:338.607 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-11-27 Thu 01:41:20:338.615 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-11-27 Thu 01:41:20:338.703 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: SANDESH: Send FAILED: 1764207680338562 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 4701 calls = 3 average_bytes = 1567 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 128 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 149, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 29, _iter59->first = sent _iter59->second = 3, ] ] ] 2025-11-27 Thu 01:41:20:338.727 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-11-27 Thu 01:41:20:341.681 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:39312::10.0.0.129:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 281 2025-11-27 Thu 01:41:20:341.706 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: 10.0.0.129:39312::10.0.0.129:8086(-1) OnSessionEvent TCP Connected 2025-11-27 Thu 01:41:20:341.713 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: 10.0.0.129:39312::10.0.0.129:8086(-1):EvTcpConnected 2025-11-27 Thu 01:41:20:341.784 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-11-27 Thu 01:41:20:341.814 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-11-27 Thu 01:41:20:341.857 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: ClientInit 2025-11-27 Thu 01:41:20:341.873 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-juju-k8s-1739-1:contrail-query-engine:0:Database count 4 2025-11-27 Thu 01:41:20:341.944 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-11-27 Thu 01:41:20:341.952 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-11-27 Thu 01:41:20:341.969 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: SANDESH: Send FAILED: 1764207680341848 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-11-27 Thu 01:41:20:341.975 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-11-27 Thu 01:41:20:341.983 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-11-27 Thu 01:41:20:341.988 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-11-27 Thu 01:41:20:342.005 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-11-27 Thu 01:41:20:342.011 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-11-27 Thu 01:41:20:342.110 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: SANDESH: Send FAILED: 1764207680341932 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 4701 calls = 3 average_bytes = 1567 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 128 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 150, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 31, _iter59->first = sent _iter59->second = 3, ] ] ] 2025-11-27 Thu 01:41:20:342.123 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567891658304, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-11-27 Thu 01:41:20:343.000 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: 10.0.0.129:39312::10.0.0.129:8086(-1) OnSessionEvent TCP Connection Closed 2025-11-27 Thu 01:41:20:343.014 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: 10.0.0.129:39312::10.0.0.129:8086(-1):EvTcpClose 2025-11-27 Thu 01:41:20:343.048 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-11-27 Thu 01:41:20:343.056 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: ClientInit : EvTcpClose 2025-11-27 Thu 01:41:20:343.088 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Return to idle with 1000 2025-11-27 Thu 01:41:20:343.094 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Idle 2025-11-27 Thu 01:41:20:343.152 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-11-27 Thu 01:41:20:343.180 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: SANDESH: Send FAILED: 1764207680343079 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-11-27 Thu 01:41:20:343.188 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-11-27 Thu 01:41:20:343.194 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-11-27 Thu 01:41:20:343.221 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-11-27 Thu 01:41:20:343.302 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: SANDESH: Send FAILED: 1764207680343140 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 6268 calls = 4 average_bytes = 1567 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 128 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 150, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 33, _iter59->first = sent _iter59->second = 4, ] ] ] 2025-11-27 Thu 01:41:20:343.319 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-11-27 Thu 01:41:21:312.489 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-11-27 Thu 01:41:21:312.581 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: SANDESH: Send FAILED: 1764207681312208 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 table_info= [ [ _iter16->first = MessageTablev2 [ reads = 3 read_fails = 0 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], _iter16->first = ObjectValueTable [ reads = 2 read_fails = 1 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], _iter16->first = SessionTable [ reads = 1 read_fails = 1 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], _iter16->first = StatsTableByStrStrTagV3 [ reads = 0 read_fails = 2 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], _iter16->first = StatsTableV4 [ reads = 2 read_fails = 0 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], _iter16->first = SystemObjectTable [ reads = 4 read_fails = 0 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 38 write_table_fails = 0 read_table_fails = 4 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 2 available_connections = 2 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-11-27 Thu 01:41:21:312.595 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-11-27 Thu 01:41:21:313.327 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x102aaf0 2025-11-27 Thu 01:41:21:313.462 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: RAC_Connect status: 0 0x102aaf0 2025-11-27 Thu 01:41:21:313.470 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: Connected to REDIS... 2025-11-27 Thu 01:41:21:313.502 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.129 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-11-27 Thu 01:41:21:313.512 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x10dfa70 2025-11-27 Thu 01:41:21:313.590 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x10e0c30 2025-11-27 Thu 01:41:21:313.629 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x10e1710 2025-11-27 Thu 01:41:21:313.668 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x10e2230 2025-11-27 Thu 01:41:21:313.757 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: RAC_Connect status: 0 0x10dfa70 2025-11-27 Thu 01:41:21:313.764 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: Connected to REDIS... 2025-11-27 Thu 01:41:21:313.774 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.129 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-11-27 Thu 01:41:21:313.782 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: RAC_Connect status: 0 0x10e0c30 2025-11-27 Thu 01:41:21:313.786 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: Connected to REDIS... 2025-11-27 Thu 01:41:21:313.792 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.129 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-11-27 Thu 01:41:21:313.799 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: RAC_Connect status: 0 0x10e1710 2025-11-27 Thu 01:41:21:313.803 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: Connected to REDIS... 2025-11-27 Thu 01:41:21:313.809 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.129 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-11-27 Thu 01:41:21:313.815 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: RAC_Connect status: 0 0x10e2230 2025-11-27 Thu 01:41:21:313.819 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: Connected to REDIS... 2025-11-27 Thu 01:41:21:313.825 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.129 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-11-27 Thu 01:41:21:313.852 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.129:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-11-27 Thu 01:41:21:313.860 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.129:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-11-27 Thu 01:41:21:313.972 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-11-27 Thu 01:41:21:314.019 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: SANDESH: Send FAILED: 1764207681313928 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.129:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-11-27 Thu 01:41:21:314.028 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-11-27 Thu 01:41:21:314.139 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-11-27 Thu 01:41:21:314.155 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.129:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-11-27 Thu 01:41:21:314.185 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.129:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-11-27 Thu 01:41:21:314.240 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-11-27 Thu 01:41:21:314.276 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: SANDESH: Send FAILED: 1764207681314202 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.129:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-11-27 Thu 01:41:21:314.284 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568657913408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-11-27 Thu 01:41:21:314.394 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-11-27 Thu 01:41:21:314.408 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.129:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-11-27 Thu 01:41:21:314.415 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.129:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-11-27 Thu 01:41:21:314.425 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.129:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-11-27 Thu 01:41:21:314.431 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.129:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-11-27 Thu 01:41:21:314.441 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.129:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-11-27 Thu 01:41:21:314.447 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.129:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-11-27 Thu 01:41:21:314.457 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-11-27 Thu 01:41:21:314.464 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-11-27 Thu 01:41:21:314.472 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-11-27 Thu 01:41:21:343.239 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-11-27 Thu 01:41:21:343.542 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: Connect : Start Connect timer 10.0.0.129:8086 2025-11-27 Thu 01:41:21:343.646 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-11-27 Thu 01:41:21:343.670 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: SANDESH: Send FAILED: 1764207681343299 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.129:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-11-27 Thu 01:41:21:343.677 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-11-27 Thu 01:41:21:343.684 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-11-27 Thu 01:41:21:343.777 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: SANDESH: Send FAILED: 1764207681343630 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 6268 calls = 4 average_bytes = 1567 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 148 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 170, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 38, _iter59->first = sent _iter59->second = 4, ] ] ] 2025-11-27 Thu 01:41:21:343.793 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-11-27 Thu 01:41:21:344.968 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:39338::10.0.0.129:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 281 2025-11-27 Thu 01:41:21:344.990 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: 10.0.0.129:39338::10.0.0.129:8086(-1) OnSessionEvent TCP Connected 2025-11-27 Thu 01:41:21:344.997 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: 10.0.0.129:39338::10.0.0.129:8086(-1):EvTcpConnected 2025-11-27 Thu 01:41:21:345.194 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567887459904, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-11-27 Thu 01:41:21:345.208 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567887459904, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-11-27 Thu 01:41:21:345.234 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567887459904, Pid 1]: ClientInit 2025-11-27 Thu 01:41:21:345.244 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567887459904, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-juju-k8s-1739-1:contrail-query-engine:0:Database count 5 2025-11-27 Thu 01:41:21:345.303 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567887459904, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-11-27 Thu 01:41:21:345.310 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567887459904, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-11-27 Thu 01:41:21:345.326 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567887459904, Pid 1]: SANDESH: Send FAILED: 1764207681345228 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.129:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-11-27 Thu 01:41:21:345.333 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567887459904, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-11-27 Thu 01:41:21:345.339 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567887459904, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-11-27 Thu 01:41:21:345.344 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567887459904, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-11-27 Thu 01:41:21:345.361 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567887459904, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-11-27 Thu 01:41:21:345.366 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567887459904, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-11-27 Thu 01:41:21:345.449 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567887459904, Pid 1]: SANDESH: Send FAILED: 1764207681345293 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 6268 calls = 4 average_bytes = 1567 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 148 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 22 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 171, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 4, ] ] ] 2025-11-27 Thu 01:41:21:345.469 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567887459904, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-11-27 Thu 01:41:21:347.079 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568662111808, Pid 1]: OnMessage control in state: ClientInit 2025-11-27 Thu 01:41:21:347.113 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568662111808, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-11-27 Thu 01:41:21:347.120 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568662111808, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-11-27 Thu 01:41:21:347.136 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568662111808, Pid 1]: Received Ctrl Message with size 0 2025-11-27 Thu 01:41:21:347.267 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568662111808, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-juju-k8s-1739-1:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1764207454841851 collector_name = collector_ip = 10.0.0.129:8086 collector_list= [ [ (*_iter40) = 10.0.0.129:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 6268 calls = 4 average_bytes = 1567 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1764207676313579 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1764207676314234 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 148 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 22 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 171, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 4, ] ] max_sm_queue_count = 6 proxy seq 21 2025-11-27 Thu 01:41:21:347.292 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568662111808, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-11-27 Thu 01:41:21:348.989 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568662111808, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-juju-k8s-1739-1 process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.129:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.129:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.129:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 21 2025-11-27 Thu 01:41:21:349.029 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568662111808, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-11-27 Thu 01:41:21:349.092 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568662111808, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-juju-k8s-1739-1 table_info= [ [ _iter16->first = MessageTablev2 [ reads = 3 read_fails = 0 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], _iter16->first = ObjectValueTable [ reads = 2 read_fails = 1 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], _iter16->first = SessionTable [ reads = 1 read_fails = 1 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], _iter16->first = StatsTableByStrStrTagV3 [ reads = 0 read_fails = 2 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], _iter16->first = StatsTableV4 [ reads = 2 read_fails = 0 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], _iter16->first = SystemObjectTable [ reads = 4 read_fails = 0 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 38 write_table_fails = 0 read_table_fails = 4 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 2 available_connections = 2 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 1 2025-11-27 Thu 01:41:21:349.107 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568662111808, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-11-27 Thu 01:41:21:349.115 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568662111808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-11-27 Thu 01:41:21:349.140 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568662111808, Pid 1]: Established 2025-11-27 Thu 01:41:21:674.515 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:45362 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:41:25:770.683 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568687294016, Pid 1]: OnMessage control in state: Established 2025-11-27 Thu 01:41:25:770.731 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568687294016, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2025-11-27 Thu 01:41:25:770.740 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568687294016, Pid 1]: Established : EvSandeshMessageRecv 2025-11-27 Thu 01:41:25:770.765 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568687294016, Pid 1]: Received Ctrl Message with size 3 2025-11-27 Thu 01:41:25:770.777 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568687294016, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 22, total = 0 2025-11-27 Thu 01:41:25:770.787 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568687294016, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 23, total = 0 2025-11-27 Thu 01:41:25:770.797 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568687294016, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2025-11-27 Thu 01:41:25:770.815 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568687294016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-11-27 Thu 01:41:46:314.645 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-11-27 Thu 01:42:46:315.183 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567895856704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-11-27 Thu 01:43:34:554.952 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:60326 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:43:37:235.061 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:60328 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:43:41:674.705 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:60338 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:43:46:315.832 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567904253504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-11-27 Thu 01:43:59:908.869 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:40974 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:44:26:268.931 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:54872 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:44:31:605.410 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:54878 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:44:34:196.523 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:55654 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:44:37:892.555 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:55664 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:44:46:331.605 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568653715008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-11-27 Thu 01:45:46:332.143 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568710977088, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-11-27 Thu 01:46:46:332.627 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568710977088, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-11-27 Thu 01:47:46:333.287 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-11-27 Thu 01:48:03:960.425 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:49384 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:48:46:334.394 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567887459904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-11-27 Thu 01:49:04:158.459 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:57356 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:49:06:683.979 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:57368 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:49:10:665.995 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:57380 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:49:46:334.837 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567887459904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-11-27 Thu 01:50:46:335.709 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-11-27 Thu 01:51:46:336.201 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567904253504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-11-27 Thu 01:52:46:336.695 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567900055104, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-11-27 Thu 01:53:16:914.477 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:48532 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:53:46:337.661 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568691492416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-11-27 Thu 01:54:39:627.896 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:60572 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:54:42:082.599 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:60584 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:54:45:918.404 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:53062 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:54:46:338.148 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568662111808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-11-27 Thu 01:55:46:339.184 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568662111808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-11-27 Thu 01:56:46:339.651 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567900055104, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-11-27 Thu 01:57:11:899.871 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:40412 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:57:38:156.665 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:55314 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:57:46:340.121 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567895856704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-11-27 Thu 01:58:04:216.255 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:41428 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:58:46:340.662 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567895856704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-11-27 Thu 01:58:51:449.213 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:47594 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:58:54:474.023 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:34732 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255 2025-11-27 Thu 01:59:46:341.177 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140567904253504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-11-27 Thu 01:59:51:401.916 UTC cn-jenkins-deploy-platform-juju-k8s-1739-1 [Thread 140568722317184, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.129:8091::10.0.0.129:42980 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 255