2025-02-06 Thu 00:50:53:318.210 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: http-server-port 8091 2025-02-06 Thu 00:50:53:318.291 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Max-tasks 16 2025-02-06 Thu 00:50:53:318.300 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Max-slice 100 2025-02-06 Thu 00:50:53:318.306 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Collectors 10.0.0.23:8086 2025-02-06 Thu 00:50:53:318.311 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Collectors 10.0.0.41:8086 2025-02-06 Thu 00:50:53:318.316 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Collectors 10.0.0.38:8086 2025-02-06 Thu 00:50:53:318.335 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: SANDESH: ROLE : Generator 2025-02-06 Thu 00:50:53:318.341 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-06 Thu 00:50:53:318.347 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2696-1. 2025-02-06 Thu 00:50:53:318.352 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-06 Thu 00:50:53:318.357 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-06 Thu 00:50:53:318.363 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-06 Thu 00:50:53:318.368 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-06 Thu 00:50:53:318.899 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: HTTP Introspect Init 2025-02-06 Thu 00:50:53:318.992 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-06 Thu 00:50:53:319.047 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Sandesh Http Server Port: 8091 2025-02-06 Thu 00:50:53:319.082 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-06 Thu 00:50:53:319.091 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-06 Thu 00:50:53:319.166 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Idle 2025-02-06 Thu 00:50:53:319.325 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: SANDESH: No Client: 1738803053319247 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:50:53:319.445 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-06 Thu 00:50:53:319.459 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-06 Thu 00:50:53:319.472 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Cassandra Servers: 10.0.0.23:9042 10.0.0.41:9042 10.0.0.38:9042 2025-02-06 Thu 00:50:53:320.015 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: Processing scm::EvStart in state Idle 2025-02-06 Thu 00:50:53:320.225 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: Disconnect 2025-02-06 Thu 00:50:53:320.322 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-06 Thu 00:50:53:320.527 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: Connect : Start Connect timer 10.0.0.41:8086 2025-02-06 Thu 00:50:53:320.607 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:50:53:320.629 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: SANDESH: Send FAILED: 1738803053320207 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-06 Thu 00:50:53:320.638 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:50:53:320.647 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:50:53:320.734 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: SANDESH: Send FAILED: 1738803053320310 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:50:53:320.752 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:50:53:320.762 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:50:53:320.852 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: SANDESH: Send FAILED: 1738803053320377 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:50:53:320.865 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:50:53:320.875 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:50:53:320.891 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: SANDESH: Send FAILED: 1738803053320401 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-06 Thu 00:50:53:320.897 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:50:53:320.906 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:50:53:320.975 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: SANDESH: Send FAILED: 1738803053320594 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:50:53:320.987 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:50:53:327.297 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-06 Thu 00:50:53:327.322 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:50:53:327.375 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:50:53:327.539 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:50:53:327.569 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: SANDESH: Send FAILED: 1738803053327354 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-06 Thu 00:50:53:327.578 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:50:58:331.650 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-06 Thu 00:50:58:331.690 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:50:58:331.727 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:03:336.226 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-06 Thu 00:51:03:336.267 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:03:336.291 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:08:338.656 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-06 Thu 00:51:08:338.696 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:08:338.719 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:13:544.513 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:13:544.573 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:13:544.598 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:18:593.027 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:18:593.068 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:18:593.093 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:23:688.157 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:23:688.207 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:23:688.231 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:28:716.810 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:28:716.863 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:28:716.886 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 7 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:33:742.496 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:33:742.536 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:33:742.559 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 8 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:38:779.447 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:38:779.492 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:38:779.515 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 9 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:43:805.944 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-06 Thu 00:51:43:805.984 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:43:806.007 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 10 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:48:863.042 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-06 Thu 00:51:48:863.093 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 11 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-06 Thu 00:51:53:884.508 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-06 Thu 00:51:58:890.645 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-06 Thu 00:51:58:890.682 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: -(-1):EvTcpConnectFail 2025-02-06 Thu 00:51:58:890.703 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x1378620 2025-02-06 Thu 00:51:58:890.709 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:890.729 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:890.743 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x1378cd0 2025-02-06 Thu 00:51:58:890.748 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:890.757 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:890.766 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x1372070 2025-02-06 Thu 00:51:58:890.772 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:890.792 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:890.802 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x13790d0 2025-02-06 Thu 00:51:58:890.807 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:890.816 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:890.824 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x1379840 2025-02-06 Thu 00:51:58:890.830 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:890.838 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:890.847 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137a180 2025-02-06 Thu 00:51:58:890.852 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:890.860 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:890.868 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137aac0 2025-02-06 Thu 00:51:58:890.873 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:890.881 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:890.889 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137b400 2025-02-06 Thu 00:51:58:890.894 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:890.902 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:890.911 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137bd40 2025-02-06 Thu 00:51:58:890.916 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:890.927 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:890.936 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137c680 2025-02-06 Thu 00:51:58:890.941 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:890.949 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:890.957 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137cfc0 2025-02-06 Thu 00:51:58:890.962 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:890.970 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:890.978 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137d900 2025-02-06 Thu 00:51:58:890.983 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:890.991 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:890.999 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137e240 2025-02-06 Thu 00:51:58:891.004 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:891.013 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:891.021 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137f4c0 2025-02-06 Thu 00:51:58:891.026 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:891.034 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:891.042 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137eb80 2025-02-06 Thu 00:51:58:891.047 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 00:51:58:891.055 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 00:51:58:891.307 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:51:58:891.345 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SANDESH: Send FAILED: 1738803118890511 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-06 Thu 00:51:58:891.358 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:58:891.369 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-06 Thu 00:51:58:891.377 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Connect : EvTcpConnectFail 2025-02-06 Thu 00:51:58:891.510 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-06 Thu 00:51:58:891.548 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Idle 2025-02-06 Thu 00:51:58:891.619 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:891.736 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SANDESH: Send FAILED: 1738803118891464 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 51 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 52, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:51:58:891.752 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:51:58:891.763 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:891.794 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SANDESH: Send FAILED: 1738803118891491 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-06 Thu 00:51:58:891.801 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:58:891.810 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 00:51:58:891.847 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:891.960 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SANDESH: Send FAILED: 1738803118891608 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 51 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 53, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:51:58:891.974 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:51:58:892.095 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:892.150 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SANDESH: Send FAILED: 1738803118891991 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738803118891546 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738803118891845 ] max_sm_queue_count = 5 ] 2025-02-06 Thu 00:51:58:892.157 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:51:58:892.166 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:892.274 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SANDESH: Send FAILED: 1738803118892058 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 51 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 53, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 10, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:51:58:892.287 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:51:58:892.936 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:892.953 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:892.968 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:892.979 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:892.992 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:893.002 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:893.015 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:893.025 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:893.043 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:893.053 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:893.066 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:893.075 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:893.088 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:893.097 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:893.110 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:893.120 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:893.133 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:893.142 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:893.155 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:893.165 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:893.225 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:893.257 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:893.318 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: SANDESH: Send FAILED: 1738803118893199 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 00:51:58:893.337 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:58:893.349 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:893.370 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: SANDESH: Send FAILED: 1738803118893253 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.23:6379::Down.10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 00:51:58:893.378 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:58:893.260 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:893.777 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:893.840 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:893.859 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:893.871 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:893.923 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: SANDESH: Send FAILED: 1738803118893815 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.23:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 00:51:58:893.934 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:58:893.877 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:894.370 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:894.388 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:894.403 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:894.419 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:894.625 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:894.644 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:894.659 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:894.669 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:894.868 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:894.886 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:894.904 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:894.914 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:894.927 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 00:51:58:894.937 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 00:51:58:894.959 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:895.006 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:895.023 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:895.031 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 00:51:58:895.089 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: SANDESH: Send FAILED: 1738803118894981 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 00:51:58:895.098 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:58:895.054 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:58:895.291 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 00:51:59:891.660 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 00:51:59:891.888 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Connect : Start Connect timer 10.0.0.23:8086 2025-02-06 Thu 00:51:59:892.002 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:51:59:892.033 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: SANDESH: Send FAILED: 1738803119891728 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 00:51:59:892.043 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 00:51:59:892.052 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 00:51:59:892.306 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: SANDESH: Send FAILED: 1738803119891989 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 98, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:51:59:892.335 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 00:51:59:894.614 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.23:46466::10.0.0.23:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-06 Thu 00:51:59:894.638 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:46466::10.0.0.23:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 00:51:59:894.646 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:46466::10.0.0.23:8086(-1):EvTcpConnected 2025-02-06 Thu 00:51:59:895.030 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 00:51:59:895.042 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 00:51:59:895.072 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: ClientInit 2025-02-06 Thu 00:51:59:895.086 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-1.:contrail-query-engine:0:Database count 1 2025-02-06 Thu 00:51:59:895.168 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 00:51:59:895.178 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 00:51:59:895.201 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SANDESH: Send FAILED: 1738803119895065 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 00:51:59:895.209 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 00:51:59:895.218 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 00:51:59:895.225 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 00:51:59:895.236 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 00:51:59:895.243 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 00:51:59:895.354 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SANDESH: Send FAILED: 1738803119895157 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 99, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 00:51:59:895.374 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 00:51:59:897.295 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: OnMessage control in state: ClientInit 2025-02-06 Thu 00:51:59:897.334 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-06 Thu 00:51:59:897.344 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-06 Thu 00:51:59:897.369 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: Received Ctrl Message with size 0 2025-02-06 Thu 00:51:59:897.534 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738803118891546 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738803118891845 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 99, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 5 proxy seq 10 2025-02-06 Thu 00:51:59:897.561 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 00:51:59:897.587 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2025-02-06 Thu 00:51:59:897.599 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 00:51:59:897.607 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-06 Thu 00:51:59:897.614 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-06 Thu 00:51:59:897.645 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: Established 2025-02-06 Thu 00:52:11:442.522 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: OnMessage control in state: Established 2025-02-06 Thu 00:52:11:442.581 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2025-02-06 Thu 00:52:11:442.592 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Established : EvSandeshMessageRecv 2025-02-06 Thu 00:52:11:442.640 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Received Ctrl Message with size 3 2025-02-06 Thu 00:52:11:442.653 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 11, total = 0 2025-02-06 Thu 00:52:11:442.664 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 13, total = 0 2025-02-06 Thu 00:52:11:442.676 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2025-02-06 Thu 00:52:11:442.684 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-06 Thu 00:52:28:896.330 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-06 Thu 00:53:28:904.288 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-06 Thu 00:54:28:904.784 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-06 Thu 00:54:35:330.133 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.23:8091::10.0.0.23:54732 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-02-06 Thu 00:55:13:944.123 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.23:8091::10.0.0.23:39122 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-02-06 Thu 00:55:28:905.276 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-06 Thu 00:56:28:905.764 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-06 Thu 00:57:28:906.904 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023670564608, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-06 Thu 00:58:28:908.001 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-06 Thu 00:59:28:908.487 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-06 Thu 01:00:28:909.451 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-06 Thu 01:01:28:910.403 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-02-06 Thu 01:02:28:914.149 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-02-06 Thu 01:03:28:914.635 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-02-06 Thu 01:04:28:915.155 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-02-06 Thu 01:05:28:915.692 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023399671552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-02-06 Thu 01:06:28:916.271 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-02-06 Thu 01:07:28:917.238 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-02-06 Thu 01:08:28:917.958 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-02-06 Thu 01:09:28:918.452 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-02-06 Thu 01:10:28:919.594 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023437444864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-02-06 Thu 01:11:28:920.093 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023670564608, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-02-06 Thu 01:12:28:921.204 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-02-06 Thu 01:13:28:921.690 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-02-06 Thu 01:14:28:922.218 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023437444864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-02-06 Thu 01:15:28:922.686 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-02-06 Thu 01:16:28:923.143 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023670564608, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-02-06 Thu 01:17:28:923.612 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023437444864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2025-02-06 Thu 01:18:28:924.328 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2025-02-06 Thu 01:19:28:924.805 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023683159808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2025-02-06 Thu 01:20:28:925.298 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023437444864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0 2025-02-06 Thu 01:21:28:925.793 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023399671552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 30 without seqno , total = 0 2025-02-06 Thu 01:22:28:926.293 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 31 without seqno , total = 0 2025-02-06 Thu 01:23:28:927.288 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023399671552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 32 without seqno , total = 0 2025-02-06 Thu 01:24:28:927.789 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 33 without seqno , total = 0 2025-02-06 Thu 01:25:28:928.282 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 34 without seqno , total = 0 2025-02-06 Thu 01:26:28:928.807 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 35 without seqno , total = 0 2025-02-06 Thu 01:27:28:929.336 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023670564608, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 36 without seqno , total = 0 2025-02-06 Thu 01:28:28:929.891 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023678961408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 37 without seqno , total = 0 2025-02-06 Thu 01:29:28:930.475 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 38 without seqno , total = 0 2025-02-06 Thu 01:30:28:931.752 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023678961408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 39 without seqno , total = 0 2025-02-06 Thu 01:31:28:932.298 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 40 without seqno , total = 0 2025-02-06 Thu 01:32:28:932.807 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023399671552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 41 without seqno , total = 0 2025-02-06 Thu 01:33:28:933.320 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023437444864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 42 without seqno , total = 0 2025-02-06 Thu 01:34:28:933.820 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 43 without seqno , total = 0 2025-02-06 Thu 01:35:28:934.331 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023670564608, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 44 without seqno , total = 0 2025-02-06 Thu 01:36:28:934.785 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023437444864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 45 without seqno , total = 0 2025-02-06 Thu 01:37:28:935.369 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 46 without seqno , total = 0 2025-02-06 Thu 01:38:28:935.944 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 47 without seqno , total = 0 2025-02-06 Thu 01:39:28:936.540 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023399671552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 48 without seqno , total = 0 2025-02-06 Thu 01:40:28:937.247 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 49 without seqno , total = 0 2025-02-06 Thu 01:41:28:937.839 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 50 without seqno , total = 0 2025-02-06 Thu 01:42:28:938.413 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023399671552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 51 without seqno , total = 0 2025-02-06 Thu 01:43:28:938.915 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023399671552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 52 without seqno , total = 0 2025-02-06 Thu 01:44:28:939.428 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 53 without seqno , total = 0 2025-02-06 Thu 01:45:28:939.897 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 54 without seqno , total = 0 2025-02-06 Thu 01:46:28:940.437 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023395473152, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 55 without seqno , total = 0 2025-02-06 Thu 01:47:28:940.954 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023678961408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 56 without seqno , total = 0 2025-02-06 Thu 01:48:28:941.467 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023395473152, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 57 without seqno , total = 0 2025-02-06 Thu 01:49:28:941.959 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 58 without seqno , total = 0 2025-02-06 Thu 01:50:28:942.845 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 59 without seqno , total = 0 2025-02-06 Thu 01:50:39:701.257 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x137eb80 2025-02-06 Thu 01:50:39:701.315 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:50:39:701.356 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x137d900 2025-02-06 Thu 01:50:39:701.368 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:50:39:701.390 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x137e240 2025-02-06 Thu 01:50:39:701.403 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:50:39:701.422 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x137f4c0 2025-02-06 Thu 01:50:39:701.434 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:50:39:701.459 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-06 Thu 01:50:39:701.469 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x137cfc0 2025-02-06 Thu 01:50:39:701.478 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:50:39:701.745 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:50:39:701.785 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:50:39:701.801 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:50:39:702.012 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:50:39:702.035 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:50:39:757.632 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x137c680 2025-02-06 Thu 01:50:39:757.679 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:50:39:757.724 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x137b400 2025-02-06 Thu 01:50:39:757.736 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:50:39:757.759 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x137bd40 2025-02-06 Thu 01:50:39:757.771 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:50:39:757.793 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x137aac0 2025-02-06 Thu 01:50:39:757.806 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:50:39:757.833 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-06 Thu 01:50:39:757.840 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x137a180 2025-02-06 Thu 01:50:39:757.849 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:50:39:758.172 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:50:39:758.200 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:50:39:758.213 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:50:39:759.013 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:50:39:759.035 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:50:44:701.817 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:50:44:701.940 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:50:44:701.983 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:50:44:702.048 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:50:44:702.084 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:50:44:702.131 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:50:44:702.157 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:50:44:702.172 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:50:44:702.230 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:50:44:702.261 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:50:44:758.236 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:50:44:758.323 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:50:44:758.368 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:50:44:758.468 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:50:44:758.491 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:50:44:758.521 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:50:44:759.046 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:50:44:759.081 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:50:44:759.227 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:50:44:759.244 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:50:49:702.180 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:50:49:702.275 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:50:49:702.309 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:50:49:702.342 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:50:49:702.379 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:50:49:702.421 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:50:49:702.443 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:50:49:702.487 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:50:49:702.502 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:50:49:702.522 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:50:49:758.694 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:50:49:758.779 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:50:49:758.815 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:50:49:758.958 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:50:49:758.985 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:50:49:759.015 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:50:49:759.256 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:50:49:759.294 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:50:49:759.445 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:50:49:759.473 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:50:54:702.473 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:50:54:702.587 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:50:54:702.630 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:50:54:702.667 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:50:54:702.697 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:50:54:702.809 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137eb80 2025-02-06 Thu 01:50:54:702.821 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:50:54:702.854 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:50:54:702.891 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137e240 2025-02-06 Thu 01:50:54:702.897 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:50:54:702.907 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:50:54:702.915 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137d900 2025-02-06 Thu 01:50:54:702.920 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:50:54:702.928 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:50:54:702.946 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137cfc0 2025-02-06 Thu 01:50:54:702.951 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:50:54:702.959 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:50:54:702.967 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137f4c0 2025-02-06 Thu 01:50:54:702.972 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:50:54:702.980 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:50:54:703.373 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:50:54:703.389 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:50:54:703.415 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:50:54:703.429 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:50:54:703.439 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:50:54:703.451 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:50:54:703.460 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:50:54:703.472 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:50:54:703.482 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:50:54:703.494 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:50:54:703.503 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:50:54:703.570 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:50:54:703.595 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:50:54:703.612 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:50:54:703.630 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:50:54:759.016 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:50:54:759.111 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:50:54:759.149 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:50:54:759.404 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:50:54:759.449 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:50:54:759.473 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:50:54:759.489 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:50:54:759.536 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:50:54:759.747 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:50:54:759.767 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:50:59:760.119 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:50:59:760.222 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:50:59:760.259 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:50:59:760.289 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:50:59:760.320 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:50:59:764.101 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137c680 2025-02-06 Thu 01:50:59:764.122 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:50:59:764.145 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:50:59:764.158 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137b400 2025-02-06 Thu 01:50:59:764.164 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:50:59:764.173 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:50:59:764.190 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137bd40 2025-02-06 Thu 01:50:59:764.196 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:50:59:764.204 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:50:59:764.212 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137aac0 2025-02-06 Thu 01:50:59:764.217 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:50:59:764.225 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:50:59:764.233 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137a180 2025-02-06 Thu 01:50:59:764.238 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:50:59:764.246 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:50:59:766.310 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:50:59:766.338 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:50:59:766.354 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:50:59:766.363 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:50:59:766.376 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:50:59:766.385 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:50:59:766.398 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:50:59:766.408 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:50:59:766.420 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:50:59:766.429 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:50:59:766.553 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:50:59:766.616 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:50:59:766.632 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:50:59:766.646 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:50:59:766.659 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:51:28:943.330 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 60 without seqno , total = 0 2025-02-06 Thu 01:52:28:949.762 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023399671552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 61 without seqno , total = 0 2025-02-06 Thu 01:53:28:950.305 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 62 without seqno , total = 0 2025-02-06 Thu 01:53:29:878.706 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-06 Thu 01:53:29:878.780 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x137a180 2025-02-06 Thu 01:53:29:878.801 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:878.859 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x137bd40 2025-02-06 Thu 01:53:29:878.872 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:878.898 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x137aac0 2025-02-06 Thu 01:53:29:878.909 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:878.933 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x137c680 2025-02-06 Thu 01:53:29:878.974 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:879.010 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x137b400 2025-02-06 Thu 01:53:29:879.021 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:879.463 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:53:29:879.532 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:53:29:879.551 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:53:29:879.564 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:53:29:879.576 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:53:29:963.672 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x1379840 2025-02-06 Thu 01:53:29:963.736 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:963.955 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x13790d0 2025-02-06 Thu 01:53:29:963.974 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:964.038 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x1372070 2025-02-06 Thu 01:53:29:964.118 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:964.152 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x1378cd0 2025-02-06 Thu 01:53:29:964.180 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:964.228 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-06 Thu 01:53:29:964.251 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x1378620 2025-02-06 Thu 01:53:29:964.261 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:29:964.604 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1379840 2025-02-06 Thu 01:53:29:964.647 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x13790d0 2025-02-06 Thu 01:53:29:964.662 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1372070 2025-02-06 Thu 01:53:29:964.688 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378cd0 2025-02-06 Thu 01:53:29:964.702 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378620 2025-02-06 Thu 01:53:29:966.979 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:46466::10.0.0.23:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-06 Thu 01:53:29:967.002 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:46466::10.0.0.23:8086(-1):EvTcpClose 2025-02-06 Thu 01:53:29:979.819 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Processing scm::EvTcpClose in state Established 2025-02-06 Thu 01:53:29:979.881 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Established : EvTcpClose 2025-02-06 Thu 01:53:29:980.153 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2025-02-06 Thu 01:53:29:980.245 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:29:980.435 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: SANDESH: Send FAILED: 1738806809979973 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = cn-jenkins-deploy-platform-ansible-os-2696-1. collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3010634 calls = 319 average_bytes = 9437 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 160 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 165, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 320, ] ] ] 2025-02-06 Thu 01:53:29:980.455 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:29:980.466 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:29:980.492 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: SANDESH: Send FAILED: 1738806809980009 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.23:6379::Down.10.0.0.41:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:53:29:980.502 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:29:980.511 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2025-02-06 Thu 01:53:29:980.533 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:29:980.678 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: SANDESH: Send FAILED: 1738806809980233 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3010634 calls = 319 average_bytes = 9437 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 160 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 165, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 320, ] ] ] 2025-02-06 Thu 01:53:29:980.692 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:29:982.558 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.23:49024::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-06 Thu 01:53:29:982.584 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:49024::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 01:53:29:982.593 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:49024::10.0.0.38:8086(-1):EvTcpConnected 2025-02-06 Thu 01:53:29:985.243 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023678961408, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 01:53:29:985.269 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023678961408, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 01:53:29:985.307 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023678961408, Pid 1]: ClientInit 2025-02-06 Thu 01:53:29:985.334 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023678961408, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-1.:contrail-query-engine:0:Database count 2 2025-02-06 Thu 01:53:29:985.426 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023678961408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:29:985.437 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023678961408, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:53:29:985.464 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023678961408, Pid 1]: SANDESH: Send FAILED: 1738806809985299 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.23:6379::Down.10.0.0.41:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:53:29:985.472 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023678961408, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:53:29:985.482 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023678961408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:29:985.489 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023678961408, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 01:53:29:985.501 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023678961408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:29:985.508 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023678961408, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 01:53:29:985.666 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023678961408, Pid 1]: SANDESH: Send FAILED: 1738806809985414 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3010634 calls = 319 average_bytes = 9437 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 160 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 166, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 320, ] ] ] 2025-02-06 Thu 01:53:29:985.688 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023678961408, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 01:53:30:117.737 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x137d900 2025-02-06 Thu 01:53:30:119.815 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:30:119.888 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x137f4c0 2025-02-06 Thu 01:53:30:119.902 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:30:119.932 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-06 Thu 01:53:30:119.941 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x137cfc0 2025-02-06 Thu 01:53:30:119.951 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:30:120.001 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x137eb80 2025-02-06 Thu 01:53:30:120.012 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:30:120.047 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Disconnect status: -1 0x137e240 2025-02-06 Thu 01:53:30:120.057 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-06 Thu 01:53:30:120.545 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:53:30:120.589 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:53:30:120.622 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:53:30:120.643 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:53:30:120.697 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:30:120.711 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:53:30:120.744 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: SANDESH: Send FAILED: 1738806810120181 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:30:120.753 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:53:30:122.802 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:53:30:141.273 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:49024::10.0.0.38:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-06 Thu 01:53:30:141.297 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:49024::10.0.0.38:8086(-1):EvTcpClose 2025-02-06 Thu 01:53:30:141.357 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-06 Thu 01:53:30:141.370 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: ClientInit : EvTcpClose 2025-02-06 Thu 01:53:30:141.508 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Return to idle with 1000 2025-02-06 Thu 01:53:30:141.518 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Idle 2025-02-06 Thu 01:53:30:141.582 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:30:141.748 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SANDESH: Send FAILED: 1738806810141463 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3012207 calls = 320 average_bytes = 9413 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 172, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 26, _iter59->first = sent _iter59->second = 321, ] ] ] 2025-02-06 Thu 01:53:30:141.772 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:30:141.783 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:30:141.808 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SANDESH: Send FAILED: 1738806810141494 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:30:141.816 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:30:141.825 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:53:30:141.861 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:30:142.008 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SANDESH: Send FAILED: 1738806810141571 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 2 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3012207 calls = 320 average_bytes = 9413 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 172, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 26, _iter59->first = sent _iter59->second = 321, ] ] ] 2025-02-06 Thu 01:53:30:142.024 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:31:141.625 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:53:31:141.866 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Connect : Start Connect timer 10.0.0.41:8086 2025-02-06 Thu 01:53:31:141.982 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:31:142.046 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SANDESH: Send FAILED: 1738806811141702 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:31:142.058 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:31:142.068 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:31:142.234 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SANDESH: Send FAILED: 1738806811141968 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3012207 calls = 320 average_bytes = 9413 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 172, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 29, _iter59->first = sent _iter59->second = 321, ] ] ] 2025-02-06 Thu 01:53:31:142.259 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:31:145.375 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.23:48088::10.0.0.41:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-06 Thu 01:53:31:145.396 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:48088::10.0.0.41:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 01:53:31:145.405 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:48088::10.0.0.41:8086(-1):EvTcpConnected 2025-02-06 Thu 01:53:31:147.247 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 01:53:31:147.264 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 01:53:31:147.299 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: ClientInit 2025-02-06 Thu 01:53:31:147.314 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-1.:contrail-query-engine:0:Database count 3 2025-02-06 Thu 01:53:31:147.400 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:31:147.409 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:53:31:147.433 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: SANDESH: Send FAILED: 1738806811147292 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:31:147.448 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:53:31:147.458 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:31:147.465 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 01:53:31:147.476 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:31:147.482 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 01:53:31:147.632 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: SANDESH: Send FAILED: 1738806811147388 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3012207 calls = 320 average_bytes = 9413 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 173, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 31, _iter59->first = sent _iter59->second = 321, ] ] ] 2025-02-06 Thu 01:53:31:147.646 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 01:53:31:150.162 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:48088::10.0.0.41:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-06 Thu 01:53:31:150.178 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:48088::10.0.0.41:8086(-1):EvTcpClose 2025-02-06 Thu 01:53:31:152.427 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-06 Thu 01:53:31:152.444 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: ClientInit : EvTcpClose 2025-02-06 Thu 01:53:31:152.564 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Return to idle with 1000 2025-02-06 Thu 01:53:31:152.573 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Idle 2025-02-06 Thu 01:53:31:152.636 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:31:152.801 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SANDESH: Send FAILED: 1738806811152517 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3013780 calls = 321 average_bytes = 9388 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 173, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 33, _iter59->first = sent _iter59->second = 322, ] ] ] 2025-02-06 Thu 01:53:31:152.819 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:31:152.829 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:31:152.866 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SANDESH: Send FAILED: 1738806811152541 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:31:152.882 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:31:152.893 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:53:31:152.902 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:31:153.045 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SANDESH: Send FAILED: 1738806811152626 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3013780 calls = 321 average_bytes = 9388 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 173, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 33, _iter59->first = sent _iter59->second = 322, ] ] ] 2025-02-06 Thu 01:53:31:153.058 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:32:155.026 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:53:32:155.311 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Connect : Start Connect timer 10.0.0.23:8086 2025-02-06 Thu 01:53:32:155.435 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:32:155.468 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: SANDESH: Send FAILED: 1738806812155109 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:32:155.478 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:32:155.488 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:32:158.038 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: SANDESH: Send FAILED: 1738806812155412 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3013780 calls = 321 average_bytes = 9388 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 173, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 36, _iter59->first = sent _iter59->second = 322, ] ] ] 2025-02-06 Thu 01:53:32:158.071 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:32:160.236 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.23:60528::10.0.0.23:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-06 Thu 01:53:32:160.255 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:60528::10.0.0.23:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 01:53:32:160.264 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:60528::10.0.0.23:8086(-1):EvTcpConnected 2025-02-06 Thu 01:53:32:173.995 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 01:53:32:174.027 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 01:53:32:174.071 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: ClientInit 2025-02-06 Thu 01:53:32:174.088 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-1.:contrail-query-engine:0:Database count 4 2025-02-06 Thu 01:53:32:174.194 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:32:174.204 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:53:32:174.231 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: SANDESH: Send FAILED: 1738806812174061 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:32:174.239 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:53:32:174.249 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:32:174.257 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 01:53:32:174.268 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:32:174.275 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 01:53:32:174.428 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: SANDESH: Send FAILED: 1738806812174182 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3013780 calls = 321 average_bytes = 9388 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 174, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 38, _iter59->first = sent _iter59->second = 322, ] ] ] 2025-02-06 Thu 01:53:32:174.451 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 01:53:32:175.926 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:60528::10.0.0.23:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-06 Thu 01:53:32:175.942 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:60528::10.0.0.23:8086(-1):EvTcpClose 2025-02-06 Thu 01:53:32:178.082 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-06 Thu 01:53:32:178.095 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: ClientInit : EvTcpClose 2025-02-06 Thu 01:53:32:178.226 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Return to idle with 1000 2025-02-06 Thu 01:53:32:178.235 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Idle 2025-02-06 Thu 01:53:32:178.301 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:32:178.459 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SANDESH: Send FAILED: 1738806812178174 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3015353 calls = 322 average_bytes = 9364 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 174, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 323, ] ] ] 2025-02-06 Thu 01:53:32:178.482 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:32:178.493 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:32:178.517 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SANDESH: Send FAILED: 1738806812178200 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:32:178.525 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:32:178.534 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:53:32:178.543 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:32:178.685 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SANDESH: Send FAILED: 1738806812178290 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3015353 calls = 322 average_bytes = 9364 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 174, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 323, ] ] ] 2025-02-06 Thu 01:53:32:178.702 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:33:184.904 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:53:33:185.142 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2025-02-06 Thu 01:53:33:185.270 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:33:185.304 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: SANDESH: Send FAILED: 1738806813184993 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:33:185.314 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:33:185.323 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:33:185.530 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: SANDESH: Send FAILED: 1738806813185254 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3015353 calls = 322 average_bytes = 9364 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 174, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 43, _iter59->first = sent _iter59->second = 323, ] ] ] 2025-02-06 Thu 01:53:33:185.548 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:33:189.964 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.23:51694::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-06 Thu 01:53:33:189.986 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:51694::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 01:53:33:189.994 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:51694::10.0.0.38:8086(-1):EvTcpConnected 2025-02-06 Thu 01:53:33:192.530 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 01:53:33:192.544 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 01:53:33:192.580 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: ClientInit 2025-02-06 Thu 01:53:33:192.596 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-1.:contrail-query-engine:0:Database count 5 2025-02-06 Thu 01:53:33:192.689 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:33:192.699 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:53:33:192.723 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SANDESH: Send FAILED: 1738806813192572 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:33:192.731 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:53:33:192.740 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:33:192.747 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 01:53:33:192.758 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:33:192.765 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 01:53:33:192.924 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SANDESH: Send FAILED: 1738806813192677 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3015353 calls = 322 average_bytes = 9364 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 175, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 45, _iter59->first = sent _iter59->second = 323, ] ] ] 2025-02-06 Thu 01:53:33:192.941 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 01:53:33:196.282 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:51694::10.0.0.38:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-06 Thu 01:53:33:196.304 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:51694::10.0.0.38:8086(-1):EvTcpClose 2025-02-06 Thu 01:53:33:198.986 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-06 Thu 01:53:33:198.998 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: ClientInit : EvTcpClose 2025-02-06 Thu 01:53:33:199.113 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Return to idle with 1000 2025-02-06 Thu 01:53:33:199.122 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Idle 2025-02-06 Thu 01:53:33:199.186 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:33:199.342 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SANDESH: Send FAILED: 1738806813199067 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3016926 calls = 323 average_bytes = 9340 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 175, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 47, _iter59->first = sent _iter59->second = 324, ] ] ] 2025-02-06 Thu 01:53:33:199.363 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:33:199.374 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:33:199.397 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SANDESH: Send FAILED: 1738806813199091 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:33:199.404 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:33:199.413 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:53:33:199.422 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:33:199.565 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: SANDESH: Send FAILED: 1738806813199176 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3016926 calls = 323 average_bytes = 9340 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 175, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 47, _iter59->first = sent _iter59->second = 324, ] ] ] 2025-02-06 Thu 01:53:33:199.581 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023445841664, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:34:199.262 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:53:34:199.474 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Connect : Start Connect timer 10.0.0.41:8086 2025-02-06 Thu 01:53:34:199.610 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:34:199.643 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: SANDESH: Send FAILED: 1738806814199343 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:34:199.653 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:34:199.662 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:34:199.860 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: SANDESH: Send FAILED: 1738806814199594 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3016926 calls = 323 average_bytes = 9340 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 175, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 50, _iter59->first = sent _iter59->second = 324, ] ] ] 2025-02-06 Thu 01:53:34:199.882 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:34:201.852 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.23:37808::10.0.0.41:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-06 Thu 01:53:34:201.883 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:37808::10.0.0.41:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 01:53:34:201.891 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:37808::10.0.0.41:8086(-1):EvTcpConnected 2025-02-06 Thu 01:53:34:201.943 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 01:53:34:201.958 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 01:53:34:201.988 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: ClientInit 2025-02-06 Thu 01:53:34:202.006 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-1.:contrail-query-engine:0:Database count 6 2025-02-06 Thu 01:53:34:202.083 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:34:202.093 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:53:34:202.116 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: SANDESH: Send FAILED: 1738806814201981 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:34:202.124 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:53:34:202.133 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:34:202.140 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 01:53:34:202.156 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:53:34:202.162 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 01:53:34:202.315 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: SANDESH: Send FAILED: 1738806814202073 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3016926 calls = 323 average_bytes = 9340 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 176, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 52, _iter59->first = sent _iter59->second = 324, ] ] ] 2025-02-06 Thu 01:53:34:202.328 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 01:53:34:879.587 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:53:34:879.712 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:53:34:879.759 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:53:34:879.793 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:53:34:879.874 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:53:34:879.954 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:53:34:879.986 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:53:34:880.012 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:53:34:880.037 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:53:34:880.051 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:53:34:964.650 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1379840 2025-02-06 Thu 01:53:34:964.749 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1379840 2025-02-06 Thu 01:53:34:964.770 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x13790d0 2025-02-06 Thu 01:53:34:964.804 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1372070 2025-02-06 Thu 01:53:34:964.833 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378cd0 2025-02-06 Thu 01:53:34:964.880 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378620 2025-02-06 Thu 01:53:34:964.926 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x13790d0 2025-02-06 Thu 01:53:34:964.948 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1372070 2025-02-06 Thu 01:53:34:964.968 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378cd0 2025-02-06 Thu 01:53:34:964.987 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378620 2025-02-06 Thu 01:53:35:120.587 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:53:35:120.670 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:53:35:120.705 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:53:35:120.733 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:53:35:120.866 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:53:35:120.901 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:53:35:120.922 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:53:35:120.936 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:53:35:122.828 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:53:35:123.037 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:53:39:880.024 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:53:39:880.150 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:53:39:880.188 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:53:39:880.227 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:53:39:880.271 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:53:39:880.321 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:53:39:880.358 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:53:39:880.373 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:53:39:880.385 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:53:39:880.471 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:53:39:964.816 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1379840 2025-02-06 Thu 01:53:39:964.932 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1379840 2025-02-06 Thu 01:53:39:964.973 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x13790d0 2025-02-06 Thu 01:53:39:965.018 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1372070 2025-02-06 Thu 01:53:39:965.058 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x13790d0 2025-02-06 Thu 01:53:39:965.074 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1372070 2025-02-06 Thu 01:53:39:965.085 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378cd0 2025-02-06 Thu 01:53:39:965.117 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378620 2025-02-06 Thu 01:53:39:965.149 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378cd0 2025-02-06 Thu 01:53:39:965.163 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378620 2025-02-06 Thu 01:53:40:120.940 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:53:40:121.045 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:53:40:121.090 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:53:40:121.123 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:53:40:121.252 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:53:40:121.278 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:53:40:121.293 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:53:40:121.310 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:53:40:123.068 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:53:40:123.239 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:53:42:713.461 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.23:37808::10.0.0.41:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2025-02-06 Thu 01:53:42:713.544 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.23:37808::10.0.0.41:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-06 Thu 01:53:42:713.568 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:37808::10.0.0.41:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-06 Thu 01:53:42:713.577 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:37808::10.0.0.41:8086(-1):EvTcpClose 2025-02-06 Thu 01:53:42:713.680 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-06 Thu 01:53:42:713.731 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: ClientInit : EvTcpClose 2025-02-06 Thu 01:53:42:714.592 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: Return to idle with 1000 2025-02-06 Thu 01:53:42:714.609 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: Idle 2025-02-06 Thu 01:53:42:714.677 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:42:714.890 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: SANDESH: Send FAILED: 1738806822714534 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 178, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 54, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-06 Thu 01:53:42:715.035 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:42:715.056 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:42:715.083 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: SANDESH: Send FAILED: 1738806822714570 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:42:715.092 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:42:715.101 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:53:42:715.133 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:42:715.285 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: SANDESH: Send FAILED: 1738806822714666 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 178, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 54, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-06 Thu 01:53:42:715.304 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:43:715.271 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023395473152, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:53:43:715.526 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023395473152, Pid 1]: Connect : Start Connect timer 10.0.0.23:8086 2025-02-06 Thu 01:53:43:715.665 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023395473152, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:43:715.699 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023395473152, Pid 1]: SANDESH: Send FAILED: 1738806823715360 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:43:715.709 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023395473152, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:43:715.721 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023395473152, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:53:43:715.892 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023395473152, Pid 1]: SANDESH: Send FAILED: 1738806823715649 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 178, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 57, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-06 Thu 01:53:43:715.918 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023395473152, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:43:716.132 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-06 Thu 01:53:43:716.174 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: -(-1):EvTcpConnectFail 2025-02-06 Thu 01:53:43:716.236 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-06 Thu 01:53:43:716.268 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Connect : EvTcpConnectFail 2025-02-06 Thu 01:53:43:716.427 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-06 Thu 01:53:43:716.469 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Idle 2025-02-06 Thu 01:53:43:716.539 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:43:716.704 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: SANDESH: Send FAILED: 1738806823716359 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 178, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 59, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-06 Thu 01:53:43:716.729 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:43:716.740 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:43:716.765 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: SANDESH: Send FAILED: 1738806823716401 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:53:43:716.772 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:53:43:716.781 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:53:43:716.809 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:43:716.967 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: SANDESH: Send FAILED: 1738806823716527 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 179, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 59, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-06 Thu 01:53:43:716.984 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023666366208, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:44:880.396 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:53:44:880.535 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:53:44:880.576 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:53:44:880.609 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:53:44:880.643 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:53:44:880.793 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:53:44:880.913 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:53:44:880.935 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:53:44:880.950 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:53:44:880.965 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:53:44:965.098 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1379840 2025-02-06 Thu 01:53:44:965.238 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1379840 2025-02-06 Thu 01:53:44:965.260 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x13790d0 2025-02-06 Thu 01:53:44:965.296 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1372070 2025-02-06 Thu 01:53:44:965.328 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378cd0 2025-02-06 Thu 01:53:44:965.356 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378620 2025-02-06 Thu 01:53:44:965.392 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x13790d0 2025-02-06 Thu 01:53:44:965.408 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1372070 2025-02-06 Thu 01:53:44:965.427 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378cd0 2025-02-06 Thu 01:53:44:965.440 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378620 2025-02-06 Thu 01:53:45:121.320 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:53:45:121.433 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:53:45:121.479 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:53:45:121.511 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:53:45:121.648 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:53:45:121.676 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:53:45:121.691 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:53:45:121.704 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:53:45:123.291 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:53:45:123.473 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:53:49:880.897 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:53:49:881.050 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:53:49:881.088 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:53:49:881.121 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:53:49:881.153 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:53:49:881.236 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:53:49:881.266 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:53:49:881.281 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:53:49:881.293 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:53:49:881.327 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:53:49:965.283 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1379840 2025-02-06 Thu 01:53:49:965.384 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1379840 2025-02-06 Thu 01:53:49:965.439 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x13790d0 2025-02-06 Thu 01:53:49:965.479 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1372070 2025-02-06 Thu 01:53:49:965.546 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x13790d0 2025-02-06 Thu 01:53:49:965.563 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1372070 2025-02-06 Thu 01:53:49:965.573 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378cd0 2025-02-06 Thu 01:53:49:965.601 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378620 2025-02-06 Thu 01:53:49:965.633 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378cd0 2025-02-06 Thu 01:53:49:965.646 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378620 2025-02-06 Thu 01:53:50:121.697 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:53:50:121.789 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:53:50:121.827 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:53:50:121.869 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:53:50:121.965 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:53:50:121.990 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:53:50:122.006 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:53:50:122.067 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:53:50:123.503 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:53:50:123.663 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:53:54:881.309 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:53:54:881.431 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:53:54:881.476 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:53:54:881.507 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:53:54:881.539 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:53:54:881.585 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:53:54:881.611 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:53:54:881.627 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:53:54:881.682 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:53:54:881.699 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:53:54:965.455 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1379840 2025-02-06 Thu 01:53:54:965.597 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1379840 2025-02-06 Thu 01:53:54:965.621 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x13790d0 2025-02-06 Thu 01:53:54:965.662 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1372070 2025-02-06 Thu 01:53:54:965.706 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x13790d0 2025-02-06 Thu 01:53:54:965.721 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1372070 2025-02-06 Thu 01:53:54:965.731 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378cd0 2025-02-06 Thu 01:53:54:965.766 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378620 2025-02-06 Thu 01:53:54:965.806 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378cd0 2025-02-06 Thu 01:53:54:965.820 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378620 2025-02-06 Thu 01:53:55:122.027 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:53:55:122.143 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:53:55:122.189 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:53:55:122.220 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:53:55:122.302 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:53:55:122.344 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:53:55:122.360 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:53:55:122.375 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:53:55:123.695 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:53:55:123.876 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:53:58:950.628 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023395473152, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:58:950.746 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023395473152, Pid 1]: SANDESH: Send FAILED: 1738806838950474 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 387 dequeues = 387 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 419 dequeues = 419 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738806823716467 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738806823716808 ] max_sm_queue_count = 5 ] 2025-02-06 Thu 01:53:58:950.773 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023395473152, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:58:950.785 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023395473152, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:53:58:950.956 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023395473152, Pid 1]: SANDESH: Send FAILED: 1738806838950582 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 179, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 62, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-06 Thu 01:53:58:950.971 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023395473152, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:53:59:881.657 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:53:59:881.786 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:53:59:881.853 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:53:59:881.902 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:53:59:881.934 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:53:59:881.984 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:53:59:882.012 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:53:59:882.028 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:53:59:882.081 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:53:59:882.098 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:53:59:965.640 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1379840 2025-02-06 Thu 01:53:59:965.752 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1379840 2025-02-06 Thu 01:53:59:965.774 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x13790d0 2025-02-06 Thu 01:53:59:965.832 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1372070 2025-02-06 Thu 01:53:59:965.879 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x13790d0 2025-02-06 Thu 01:53:59:965.895 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1372070 2025-02-06 Thu 01:53:59:965.905 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378cd0 2025-02-06 Thu 01:53:59:965.933 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378620 2025-02-06 Thu 01:53:59:965.966 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378cd0 2025-02-06 Thu 01:53:59:965.980 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378620 2025-02-06 Thu 01:54:00:122.344 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:54:00:122.428 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:54:00:122.464 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:54:00:122.495 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:54:00:122.695 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:54:00:122.718 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:54:00:122.731 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:54:00:122.754 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:54:00:123.899 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:54:00:124.091 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:54:03:928.428 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:03:928.524 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: SANDESH: Send FAILED: 1738806843928346 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-02-06 Thu 01:54:03:928.537 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023687358208, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-02-06 Thu 01:54:04:882.062 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:54:04:882.216 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:54:04:882.266 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:54:04:882.297 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:54:04:882.329 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:54:04:882.420 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:54:04:882.451 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:54:04:882.465 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:54:04:882.478 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:54:04:882.504 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:54:04:965.806 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1379840 2025-02-06 Thu 01:54:04:965.982 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1379840 2025-02-06 Thu 01:54:04:966.008 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x13790d0 2025-02-06 Thu 01:54:04:966.042 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1372070 2025-02-06 Thu 01:54:04:966.079 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x13790d0 2025-02-06 Thu 01:54:04:966.093 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1372070 2025-02-06 Thu 01:54:04:966.116 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378cd0 2025-02-06 Thu 01:54:04:966.145 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378620 2025-02-06 Thu 01:54:04:966.176 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378cd0 2025-02-06 Thu 01:54:04:966.190 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378620 2025-02-06 Thu 01:54:05:122.768 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:54:05:122.940 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:54:05:122.992 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:54:05:123.022 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:54:05:123.160 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:54:05:123.190 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:54:05:123.203 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:54:05:123.217 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:54:05:124.117 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:54:05:124.305 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:54:09:882.491 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:54:09:882.611 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:54:09:882.647 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:54:09:882.678 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:54:09:882.708 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:54:09:882.784 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:54:09:882.810 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:54:09:882.835 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:54:09:882.859 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:54:09:882.885 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:54:09:966.031 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1379840 2025-02-06 Thu 01:54:09:966.157 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1379840 2025-02-06 Thu 01:54:09:966.199 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x13790d0 2025-02-06 Thu 01:54:09:966.236 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1372070 2025-02-06 Thu 01:54:09:966.276 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x13790d0 2025-02-06 Thu 01:54:09:966.292 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1372070 2025-02-06 Thu 01:54:09:966.302 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378cd0 2025-02-06 Thu 01:54:09:966.331 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378620 2025-02-06 Thu 01:54:09:966.364 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378cd0 2025-02-06 Thu 01:54:09:966.378 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378620 2025-02-06 Thu 01:54:10:123.258 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:54:10:123.445 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:54:10:123.496 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:54:10:123.568 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:54:10:123.809 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:54:10:123.875 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:54:10:123.899 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:54:10:123.920 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:54:10:124.333 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:54:10:124.535 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:54:13:716.585 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:54:13:716.799 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2025-02-06 Thu 01:54:13:716.913 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:54:13:716.946 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: SANDESH: Send FAILED: 1738806853716674 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:54:13:716.963 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:54:13:716.972 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:54:13:717.131 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: SANDESH: Send FAILED: 1738806853716900 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 179, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 65, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-06 Thu 01:54:13:717.146 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:13:717.193 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-06 Thu 01:54:13:717.203 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: -(-1):EvTcpConnectFail 2025-02-06 Thu 01:54:13:717.258 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-06 Thu 01:54:13:717.291 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Connect : EvTcpConnectFail 2025-02-06 Thu 01:54:13:717.436 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-06 Thu 01:54:13:717.475 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Idle 2025-02-06 Thu 01:54:13:717.547 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:13:717.709 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SANDESH: Send FAILED: 1738806853717380 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 179, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 67, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-06 Thu 01:54:13:717.730 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:13:717.741 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:13:717.765 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SANDESH: Send FAILED: 1738806853717409 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:54:13:717.775 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:54:13:717.784 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:54:13:717.809 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:13:717.968 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SANDESH: Send FAILED: 1738806853717536 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 180, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 67, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-06 Thu 01:54:13:717.982 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:14:882.871 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:54:14:882.976 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:54:14:883.019 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:54:14:883.050 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:54:14:883.081 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:54:14:883.227 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:54:14:883.265 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:54:14:883.279 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:54:14:883.293 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:54:14:883.305 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:54:14:966.220 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1379840 2025-02-06 Thu 01:54:14:966.340 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1379840 2025-02-06 Thu 01:54:14:966.362 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x13790d0 2025-02-06 Thu 01:54:14:966.406 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1372070 2025-02-06 Thu 01:54:14:966.444 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x13790d0 2025-02-06 Thu 01:54:14:966.459 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1372070 2025-02-06 Thu 01:54:14:966.469 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378cd0 2025-02-06 Thu 01:54:14:966.497 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378620 2025-02-06 Thu 01:54:14:966.528 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378cd0 2025-02-06 Thu 01:54:14:966.542 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378620 2025-02-06 Thu 01:54:15:123.899 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:54:15:124.034 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:54:15:124.094 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:54:15:124.127 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:54:15:124.245 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:54:15:124.285 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:54:15:124.300 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:54:15:124.313 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:54:15:124.562 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:54:15:124.754 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:54:19:883.314 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:54:19:883.460 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:54:19:883.501 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:54:19:883.533 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:54:19:883.565 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:54:19:883.695 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:54:19:883.747 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:54:19:883.761 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:54:19:883.775 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:54:19:892.279 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:54:19:966.402 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1379840 2025-02-06 Thu 01:54:19:966.557 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1379840 2025-02-06 Thu 01:54:19:966.581 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x13790d0 2025-02-06 Thu 01:54:19:966.616 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1372070 2025-02-06 Thu 01:54:19:966.655 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x13790d0 2025-02-06 Thu 01:54:19:966.671 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1372070 2025-02-06 Thu 01:54:19:966.681 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378cd0 2025-02-06 Thu 01:54:19:966.710 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378620 2025-02-06 Thu 01:54:19:966.741 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378cd0 2025-02-06 Thu 01:54:19:966.756 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378620 2025-02-06 Thu 01:54:20:124.333 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:54:20:124.454 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:54:20:124.497 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:54:20:124.527 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:54:20:124.679 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:54:20:124.708 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:54:20:124.731 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:54:20:124.744 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:54:20:124.781 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:54:20:124.934 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:54:24:883.792 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:54:24:883.922 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:54:24:883.965 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:54:24:883.996 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:54:24:884.163 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:54:24:884.190 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:54:24:884.205 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:54:24:884.218 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:54:24:892.328 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:54:24:892.575 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:54:24:966.639 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1379840 2025-02-06 Thu 01:54:24:966.784 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1379840 2025-02-06 Thu 01:54:24:966.809 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x13790d0 2025-02-06 Thu 01:54:24:966.856 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1372070 2025-02-06 Thu 01:54:24:966.889 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378cd0 2025-02-06 Thu 01:54:24:966.918 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378620 2025-02-06 Thu 01:54:24:966.953 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x13790d0 2025-02-06 Thu 01:54:24:966.968 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1372070 2025-02-06 Thu 01:54:24:966.981 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378cd0 2025-02-06 Thu 01:54:24:966.993 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378620 2025-02-06 Thu 01:54:25:124.752 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:54:25:124.883 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:54:25:124.930 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:54:25:124.972 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:54:25:125.011 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:54:25:125.078 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:54:25:125.108 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:54:25:125.129 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:54:25:125.168 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:54:25:125.184 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:54:28:951.303 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 63 without seqno , total = 0 2025-02-06 Thu 01:54:28:951.362 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:28:951.435 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: SANDESH: Send FAILED: 1738806868951165 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 395 dequeues = 395 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 430 dequeues = 430 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738806853717474 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738806853717807 ] max_sm_queue_count = 5 ] 2025-02-06 Thu 01:54:28:951.449 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:28:951.458 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:28:951.627 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: SANDESH: Send FAILED: 1738806868951285 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 180, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 70, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-06 Thu 01:54:28:951.651 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:29:884.239 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:54:29:884.358 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:54:29:884.401 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:54:29:884.434 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:54:29:884.574 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:54:29:884.610 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:54:29:884.624 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:54:29:884.637 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:54:29:892.617 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:54:29:892.850 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:54:29:966.833 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1379840 2025-02-06 Thu 01:54:29:966.941 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1379840 2025-02-06 Thu 01:54:29:966.984 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x13790d0 2025-02-06 Thu 01:54:29:967.037 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x13790d0 2025-02-06 Thu 01:54:29:967.051 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1372070 2025-02-06 Thu 01:54:29:967.080 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378cd0 2025-02-06 Thu 01:54:29:967.110 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378620 2025-02-06 Thu 01:54:29:967.142 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1372070 2025-02-06 Thu 01:54:29:967.156 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378cd0 2025-02-06 Thu 01:54:29:967.169 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378620 2025-02-06 Thu 01:54:30:125.156 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:54:30:125.281 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:54:30:125.325 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:54:30:125.355 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:54:30:125.385 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:54:30:125.425 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:54:30:125.449 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:54:30:125.518 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:54:30:125.533 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:54:30:125.547 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:54:34:884.646 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:54:34:884.769 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:54:34:884.808 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:54:34:884.848 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:54:34:884.946 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:54:34:885.038 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:54:34:885.055 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:54:34:885.068 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:54:34:892.896 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:54:34:893.155 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:54:34:966.989 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1379840 2025-02-06 Thu 01:54:34:967.097 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1379840 2025-02-06 Thu 01:54:34:967.118 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x13790d0 2025-02-06 Thu 01:54:34:967.181 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x13790d0 2025-02-06 Thu 01:54:34:967.195 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1372070 2025-02-06 Thu 01:54:34:967.225 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378cd0 2025-02-06 Thu 01:54:34:967.261 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1372070 2025-02-06 Thu 01:54:34:967.276 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378cd0 2025-02-06 Thu 01:54:34:967.286 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378620 2025-02-06 Thu 01:54:34:967.318 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378620 2025-02-06 Thu 01:54:35:125.470 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:54:35:125.559 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:54:35:125.594 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:54:35:125.624 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:54:35:125.658 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:54:35:125.701 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:54:35:125.723 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:54:35:125.753 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:54:35:125.768 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:54:35:125.849 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:54:39:885.014 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:54:39:885.159 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:54:39:885.192 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:54:39:885.225 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:54:39:885.389 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:54:39:885.452 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:54:39:885.502 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:54:39:885.519 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:54:39:893.225 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:54:39:893.508 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:54:39:967.160 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1379840 2025-02-06 Thu 01:54:39:967.308 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1379840 2025-02-06 Thu 01:54:39:967.343 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x13790d0 2025-02-06 Thu 01:54:39:967.381 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1372070 2025-02-06 Thu 01:54:39:967.411 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378cd0 2025-02-06 Thu 01:54:39:967.450 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x13790d0 2025-02-06 Thu 01:54:39:967.466 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1372070 2025-02-06 Thu 01:54:39:967.479 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378cd0 2025-02-06 Thu 01:54:39:967.488 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378620 2025-02-06 Thu 01:54:39:967.521 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378620 2025-02-06 Thu 01:54:40:125.775 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:54:40:125.911 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:54:40:125.954 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:54:40:125.985 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:54:40:126.014 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:54:40:126.137 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:54:40:126.171 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:54:40:126.191 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:54:40:126.206 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:54:40:126.218 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:54:43:719.470 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:54:43:719.681 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Connect : Start Connect timer 10.0.0.41:8086 2025-02-06 Thu 01:54:43:719.783 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:54:43:719.815 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: SANDESH: Send FAILED: 1738806883719554 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.41:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:54:43:719.824 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:54:43:719.834 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:54:43:720.000 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: SANDESH: Send FAILED: 1738806883719770 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.41:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 180, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 72, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-06 Thu 01:54:43:720.024 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:43:725.934 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-06 Thu 01:54:43:725.959 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: -(-1):EvTcpConnectFail 2025-02-06 Thu 01:54:43:728.466 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-06 Thu 01:54:43:728.487 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: Connect : EvTcpConnectFail 2025-02-06 Thu 01:54:43:728.631 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-06 Thu 01:54:43:728.667 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: Idle 2025-02-06 Thu 01:54:43:728.738 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:43:728.924 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: SANDESH: Send FAILED: 1738806883728575 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 180, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 74, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-06 Thu 01:54:43:728.948 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:43:728.959 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:43:728.984 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: SANDESH: Send FAILED: 1738806883728605 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:54:43:728.992 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:54:43:729.000 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:54:43:729.010 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:43:729.153 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: SANDESH: Send FAILED: 1738806883728726 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 181, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 74, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-06 Thu 01:54:43:729.168 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:44:885.920 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:54:44:886.040 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:54:44:886.075 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:54:44:886.107 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:54:44:886.211 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:54:44:886.242 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:54:44:886.257 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:54:44:886.304 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:54:44:893.557 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:54:44:893.807 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:54:44:967.357 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1379840 2025-02-06 Thu 01:54:44:967.466 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1379840 2025-02-06 Thu 01:54:44:967.495 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x13790d0 2025-02-06 Thu 01:54:44:967.539 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1372070 2025-02-06 Thu 01:54:44:967.570 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378cd0 2025-02-06 Thu 01:54:44:967.609 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x13790d0 2025-02-06 Thu 01:54:44:967.625 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1372070 2025-02-06 Thu 01:54:44:967.637 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378cd0 2025-02-06 Thu 01:54:44:967.647 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378620 2025-02-06 Thu 01:54:44:967.678 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378620 2025-02-06 Thu 01:54:45:129.860 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:54:45:129.992 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:54:45:130.030 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:54:45:130.072 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:54:45:130.103 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:54:45:130.214 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:54:45:130.244 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:54:45:130.259 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:54:45:130.273 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:54:45:130.288 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:54:49:886.292 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:54:49:886.410 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:54:49:886.459 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:54:49:886.491 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:54:49:886.662 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:54:49:886.705 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:54:49:886.721 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:54:49:886.735 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:54:49:893.862 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:54:49:894.094 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:54:49:967.512 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1379840 2025-02-06 Thu 01:54:49:967.613 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1379840 2025-02-06 Thu 01:54:49:967.652 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x13790d0 2025-02-06 Thu 01:54:49:967.697 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1372070 2025-02-06 Thu 01:54:49:967.728 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378cd0 2025-02-06 Thu 01:54:49:967.765 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x13790d0 2025-02-06 Thu 01:54:49:967.781 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1372070 2025-02-06 Thu 01:54:49:967.793 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378cd0 2025-02-06 Thu 01:54:49:967.803 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378620 2025-02-06 Thu 01:54:49:967.856 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378620 2025-02-06 Thu 01:54:50:130.288 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:54:50:130.400 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:54:50:130.443 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:54:50:130.477 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:54:50:130.508 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:54:50:130.615 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:54:50:130.644 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:54:50:130.659 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:54:50:130.677 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:54:50:130.690 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:54:54:886.741 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:54:54:886.862 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:54:54:886.901 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:54:54:886.933 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:54:54:887.011 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:54:54:887.038 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:54:54:887.054 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:54:54:887.069 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:54:54:894.135 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:54:54:894.359 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:54:54:967.674 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1379840 2025-02-06 Thu 01:54:54:967.760 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1379840 2025-02-06 Thu 01:54:54:967.796 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x13790d0 2025-02-06 Thu 01:54:54:967.847 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1372070 2025-02-06 Thu 01:54:54:967.895 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x13790d0 2025-02-06 Thu 01:54:54:967.912 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1372070 2025-02-06 Thu 01:54:54:967.922 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378cd0 2025-02-06 Thu 01:54:54:967.952 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378620 2025-02-06 Thu 01:54:54:967.983 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378cd0 2025-02-06 Thu 01:54:54:967.998 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378620 2025-02-06 Thu 01:54:55:130.677 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:54:55:130.793 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:54:55:130.843 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:54:55:130.876 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:54:55:130.906 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:54:55:130.990 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:54:55:131.019 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:54:55:131.034 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:54:55:131.049 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:54:55:131.064 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:54:58:954.269 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:58:954.364 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: SANDESH: Send FAILED: 1738806898954127 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 402 dequeues = 402 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 440 dequeues = 440 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738806883728665 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738806883729009 ] max_sm_queue_count = 5 ] 2025-02-06 Thu 01:54:58:954.387 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:58:954.398 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:54:58:954.555 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: SANDESH: Send FAILED: 1738806898954240 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 181, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 77, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-06 Thu 01:54:58:954.569 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023662167808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:54:59:887.087 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:54:59:887.214 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:54:59:887.260 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:54:59:887.293 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:54:59:887.428 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:54:59:887.458 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:54:59:887.581 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:54:59:887.601 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:54:59:894.406 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:54:59:894.671 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:54:59:967.804 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1379840 2025-02-06 Thu 01:54:59:967.937 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1379840 2025-02-06 Thu 01:54:59:967.959 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x13790d0 2025-02-06 Thu 01:54:59:968.003 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1372070 2025-02-06 Thu 01:54:59:968.044 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x13790d0 2025-02-06 Thu 01:54:59:968.091 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1372070 2025-02-06 Thu 01:54:59:968.104 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378cd0 2025-02-06 Thu 01:54:59:968.136 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378620 2025-02-06 Thu 01:54:59:968.168 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378cd0 2025-02-06 Thu 01:54:59:968.182 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378620 2025-02-06 Thu 01:55:00:131.055 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:55:00:131.155 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:55:00:131.199 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:55:00:131.232 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:55:00:131.263 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:55:00:131.446 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:55:00:131.510 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:55:00:131.529 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:55:00:131.545 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:55:00:131.560 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:55:03:931.623 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023670564608, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:03:931.707 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023670564608, Pid 1]: SANDESH: Send FAILED: 1738806903931577 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-02-06 Thu 01:55:03:931.728 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023670564608, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-02-06 Thu 01:55:04:887.981 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:55:04:888.116 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:55:04:888.158 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:55:04:888.190 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:55:04:888.311 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137bd40 2025-02-06 Thu 01:55:04:888.345 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137a180 2025-02-06 Thu 01:55:04:888.361 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137b400 2025-02-06 Thu 01:55:04:888.376 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137aac0 2025-02-06 Thu 01:55:04:894.723 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:55:04:894.961 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137c680 2025-02-06 Thu 01:55:04:967.978 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1379840 2025-02-06 Thu 01:55:04:968.114 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1379840 2025-02-06 Thu 01:55:04:968.137 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x13790d0 2025-02-06 Thu 01:55:04:968.188 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x13790d0 2025-02-06 Thu 01:55:04:968.202 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1372070 2025-02-06 Thu 01:55:04:968.239 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1372070 2025-02-06 Thu 01:55:04:968.251 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378cd0 2025-02-06 Thu 01:55:04:968.279 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378620 2025-02-06 Thu 01:55:04:968.314 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378cd0 2025-02-06 Thu 01:55:04:968.329 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x1378620 2025-02-06 Thu 01:55:05:131.565 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:55:05:131.816 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:55:05:131.877 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:55:05:131.911 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:55:05:131.942 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:55:05:132.236 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137cfc0 2025-02-06 Thu 01:55:05:132.270 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137d900 2025-02-06 Thu 01:55:05:132.284 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137f4c0 2025-02-06 Thu 01:55:05:132.297 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137eb80 2025-02-06 Thu 01:55:05:132.309 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: -1 0x137e240 2025-02-06 Thu 01:55:09:892.126 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137bd40 2025-02-06 Thu 01:55:09:892.260 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137a180 2025-02-06 Thu 01:55:09:892.297 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137b400 2025-02-06 Thu 01:55:09:892.326 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137aac0 2025-02-06 Thu 01:55:09:896.115 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137bd40 2025-02-06 Thu 01:55:09:896.136 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:896.163 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:896.176 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137b400 2025-02-06 Thu 01:55:09:896.182 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:896.191 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:896.200 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137a180 2025-02-06 Thu 01:55:09:896.205 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:896.213 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:896.222 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137aac0 2025-02-06 Thu 01:55:09:896.227 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:896.248 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:896.265 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137c680 2025-02-06 Thu 01:55:09:898.244 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137c680 2025-02-06 Thu 01:55:09:898.262 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:898.278 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:898.304 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:898.315 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:898.329 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:898.338 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:898.351 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:898.360 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:898.372 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:898.382 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:898.414 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:898.496 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:898.516 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:898.537 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:898.540 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:09:898.615 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SANDESH: Send FAILED: 1738806909898448 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.23:6379::Down.10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:09:898.636 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:55:09:901.646 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:901.663 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:901.689 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:968.172 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1379840 2025-02-06 Thu 01:55:09:968.316 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x1379840 2025-02-06 Thu 01:55:09:968.335 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:968.354 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:968.364 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x13790d0 2025-02-06 Thu 01:55:09:968.411 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1372070 2025-02-06 Thu 01:55:09:968.476 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x13790d0 2025-02-06 Thu 01:55:09:968.484 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:968.495 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:968.503 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x1372070 2025-02-06 Thu 01:55:09:968.509 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:968.517 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:968.524 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378cd0 2025-02-06 Thu 01:55:09:968.561 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1378620 2025-02-06 Thu 01:55:09:968.618 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x1378cd0 2025-02-06 Thu 01:55:09:968.632 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:968.643 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:968.651 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x1378620 2025-02-06 Thu 01:55:09:968.657 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:09:968.665 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:09:968.956 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:968.974 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:968.989 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:968.998 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:969.011 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:969.021 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:969.034 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:969.043 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:969.056 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:09:969.065 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:09:969.089 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:971.974 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023399671552, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:09:972.016 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023399671552, Pid 1]: SANDESH: Send FAILED: 1738806909969120 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:09:972.032 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023399671552, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:55:09:972.053 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:972.097 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:972.115 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:09:972.130 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:10:132.308 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137cfc0 2025-02-06 Thu 01:55:10:132.434 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137d900 2025-02-06 Thu 01:55:10:132.471 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137f4c0 2025-02-06 Thu 01:55:10:132.503 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137eb80 2025-02-06 Thu 01:55:10:132.533 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x137e240 2025-02-06 Thu 01:55:10:132.609 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137cfc0 2025-02-06 Thu 01:55:10:132.618 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:10:132.643 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:10:132.671 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137d900 2025-02-06 Thu 01:55:10:132.677 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:10:132.687 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:10:132.696 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137f4c0 2025-02-06 Thu 01:55:10:132.701 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:10:132.709 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:10:132.751 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137eb80 2025-02-06 Thu 01:55:10:132.758 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:10:132.766 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:10:132.798 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: RAC_Connect status: 0 0x137e240 2025-02-06 Thu 01:55:10:132.804 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:55:10:132.812 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:55:10:134.003 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:10:134.021 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:10:134.038 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:10:134.048 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:10:134.061 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:10:134.070 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:10:134.084 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:10:134.093 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:10:134.106 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:55:10:134.116 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:55:10:134.140 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:10:135.002 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:10:135.043 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SANDESH: Send FAILED: 1738806910134172 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:10:135.059 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:55:10:135.005 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:10:137.314 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:10:137.328 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:10:137.341 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:55:13:731.564 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:55:13:731.972 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Connect : Start Connect timer 10.0.0.23:8086 2025-02-06 Thu 01:55:13:732.107 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:55:13:732.141 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SANDESH: Send FAILED: 1738806913731814 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:13:732.151 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:55:13:732.161 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:55:13:732.322 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SANDESH: Send FAILED: 1738806913732090 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 241, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 83, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-06 Thu 01:55:13:732.346 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:55:13:734.433 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-06 Thu 01:55:13:734.452 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: -(-1):EvTcpConnectFail 2025-02-06 Thu 01:55:13:737.996 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023403869952, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-06 Thu 01:55:13:738.021 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023403869952, Pid 1]: Connect : EvTcpConnectFail 2025-02-06 Thu 01:55:13:738.156 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023403869952, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-06 Thu 01:55:13:738.189 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023403869952, Pid 1]: Idle 2025-02-06 Thu 01:55:13:738.256 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023403869952, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:13:738.413 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023403869952, Pid 1]: SANDESH: Send FAILED: 1738806913738104 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 46 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 241, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 85, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-06 Thu 01:55:13:738.434 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023403869952, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:55:13:738.445 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023403869952, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:13:738.469 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023403869952, Pid 1]: SANDESH: Send FAILED: 1738806913738130 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:13:738.476 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023403869952, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:55:13:738.485 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023403869952, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:55:13:738.495 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023403869952, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:13:738.639 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023403869952, Pid 1]: SANDESH: Send FAILED: 1738806913738246 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 46 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 242, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 85, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-06 Thu 01:55:13:738.654 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023403869952, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:55:28:958.976 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 64 without seqno , total = 0 2025-02-06 Thu 01:55:28:959.043 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:28:959.118 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SANDESH: Send FAILED: 1738806928958847 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 413 dequeues = 413 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 454 dequeues = 454 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738806913738188 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738806913738493 ] max_sm_queue_count = 5 ] 2025-02-06 Thu 01:55:28:959.132 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:55:28:959.152 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:55:28:959.318 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SANDESH: Send FAILED: 1738806928958957 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 242, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 88, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-06 Thu 01:55:28:959.333 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:55:43:738.568 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-06 Thu 01:55:43:739.572 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2025-02-06 Thu 01:55:43:739.691 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:55:43:739.725 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: SANDESH: Send FAILED: 1738806943739450 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:43:739.742 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:55:43:739.752 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:55:43:739.951 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: SANDESH: Send FAILED: 1738806943739673 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 50 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 242, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 90, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-06 Thu 01:55:43:739.970 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023441643264, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:55:43:742.186 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.23:45402::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-06 Thu 01:55:43:742.208 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:45402::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 01:55:43:742.216 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023831029888, Pid 1]: 10.0.0.23:45402::10.0.0.38:8086(-1):EvTcpConnected 2025-02-06 Thu 01:55:43:742.626 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 01:55:43:743.273 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 01:55:43:743.315 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: ClientInit 2025-02-06 Thu 01:55:43:743.332 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-1.:contrail-query-engine:0:Database count 7 2025-02-06 Thu 01:55:43:743.423 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:55:43:743.433 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:55:43:743.457 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SANDESH: Send FAILED: 1738806943743307 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:55:43:743.465 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:55:43:743.474 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:55:43:743.481 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 01:55:43:743.493 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:55:43:743.500 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 01:55:43:743.652 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: SANDESH: Send FAILED: 1738806943743412 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 7 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 51 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 243, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 92, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-06 Thu 01:55:43:743.671 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023450040064, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 01:55:43:745.622 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: OnMessage control in state: ClientInit 2025-02-06 Thu 01:55:43:748.009 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-06 Thu 01:55:43:748.032 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-06 Thu 01:55:43:748.721 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: Received Ctrl Message with size 0 2025-02-06 Thu 01:55:43:749.004 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 7 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3018499 calls = 324 average_bytes = 9316 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 413 dequeues = 413 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 454 dequeues = 454 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1738806913738188 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1738806913738493 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 316 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 2982314 calls = 315 average_bytes = 9467 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 62 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 248 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 51 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 243, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 92, _iter59->first = sent _iter59->second = 325, ] ] max_sm_queue_count = 5 proxy seq 300 2025-02-06 Thu 01:55:43:749.042 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 01:55:43:749.069 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 48 2025-02-06 Thu 01:55:43:749.093 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 01:55:43:749.129 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 64 2025-02-06 Thu 01:55:43:749.139 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 01:55:43:749.151 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-06 Thu 01:55:43:749.180 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023657969408, Pid 1]: Established 2025-02-06 Thu 01:56:28:959.947 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023403869952, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 65 without seqno , total = 0 2025-02-06 Thu 01:57:28:961.881 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023437444864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 66 without seqno , total = 0 2025-02-06 Thu 01:58:28:963.686 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023674763008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 67 without seqno , total = 0 2025-02-06 Thu 01:58:57:363.740 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: Processing scm::EvStop in state Established 2025-02-06 Thu 01:58:57:363.860 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: Idle 2025-02-06 Thu 01:58:57:363.977 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-06 Thu 01:58:57:363.990 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-06 Thu 01:58:57:364.252 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: SANDESH: Send FAILED: 1738807137363962 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 7 pid = 1 http_port = 8091 start_time = 1738803053319175 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.41:8086, (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3215277 calls = 343 average_bytes = 9373 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 66 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 262 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 52 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 243, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 94, _iter59->first = sent _iter59->second = 347, ] ] ] 2025-02-06 Thu 01:58:57:364.279 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 140023408068352, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:59:25:299.556 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: http-server-port 8091 2025-02-06 Thu 01:59:25:299.635 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Max-tasks 16 2025-02-06 Thu 01:59:25:299.644 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Max-slice 100 2025-02-06 Thu 01:59:25:299.650 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Collectors 10.0.0.23:8086 2025-02-06 Thu 01:59:25:299.655 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Collectors 10.0.0.41:8086 2025-02-06 Thu 01:59:25:299.660 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Collectors 10.0.0.38:8086 2025-02-06 Thu 01:59:25:299.685 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: SANDESH: ROLE : Generator 2025-02-06 Thu 01:59:25:299.692 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-06 Thu 01:59:25:299.698 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2696-1. 2025-02-06 Thu 01:59:25:299.703 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-06 Thu 01:59:25:299.709 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-06 Thu 01:59:25:299.714 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-06 Thu 01:59:25:299.720 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-06 Thu 01:59:25:301.358 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: HTTP Introspect Init 2025-02-06 Thu 01:59:25:301.806 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-06 Thu 01:59:25:301.994 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Sandesh Http Server Port: 8091 2025-02-06 Thu 01:59:25:302.132 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-06 Thu 01:59:25:302.142 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-06 Thu 01:59:25:302.434 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Idle 2025-02-06 Thu 01:59:25:302.776 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: SANDESH: No Client: 1738807165302674 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738807165302449 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 01:59:25:303.093 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-06 Thu 01:59:25:303.212 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-06 Thu 01:59:25:303.229 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Cassandra Servers: 10.0.0.23:9042 10.0.0.41:9042 10.0.0.38:9042 2025-02-06 Thu 01:59:25:307.980 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: Processing scm::EvStart in state Idle 2025-02-06 Thu 01:59:25:308.049 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: Disconnect 2025-02-06 Thu 01:59:25:308.153 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-06 Thu 01:59:25:308.357 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: Connect : Start Connect timer 10.0.0.23:8086 2025-02-06 Thu 01:59:25:308.431 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:59:25:308.455 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: SANDESH: Send FAILED: 1738807165308040 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-06 Thu 01:59:25:308.463 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:59:25:308.473 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:59:25:308.564 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: SANDESH: Send FAILED: 1738807165308140 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738807165302449 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 01:59:25:308.580 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:59:25:308.591 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:59:25:308.666 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: SANDESH: Send FAILED: 1738807165308219 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738807165302449 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 01:59:25:308.679 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:59:25:308.688 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:59:25:308.704 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: SANDESH: Send FAILED: 1738807165308241 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-06 Thu 01:59:25:308.711 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:59:25:308.719 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:59:25:308.790 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: SANDESH: Send FAILED: 1738807165308422 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1738807165302449 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 01:59:25:308.802 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-06 Thu 01:59:25:346.154 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-06 Thu 01:59:25:349.263 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: RAC_Connect status: 0 0x1eb8470 2025-02-06 Thu 01:59:25:349.281 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:59:25:349.297 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:59:25:349.309 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: RAC_Connect status: 0 0x1eb8ca0 2025-02-06 Thu 01:59:25:349.315 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:59:25:349.324 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:59:25:349.333 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: RAC_Connect status: 0 0x1eb9520 2025-02-06 Thu 01:59:25:349.338 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:59:25:349.347 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:59:25:349.356 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: RAC_Connect status: 0 0x1eb9da0 2025-02-06 Thu 01:59:25:349.361 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:59:25:349.374 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:59:25:349.383 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: RAC_Connect status: 0 0x1ef5760 2025-02-06 Thu 01:59:25:349.388 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:59:25:349.397 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.23 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:59:25:349.405 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: RAC_Connect status: 0 0x1ef5fc0 2025-02-06 Thu 01:59:25:349.410 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:59:25:349.418 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:59:25:349.426 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: RAC_Connect status: 0 0x1ef70c0 2025-02-06 Thu 01:59:25:349.431 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:59:25:349.439 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:59:25:349.448 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: RAC_Connect status: 0 0x1ef6840 2025-02-06 Thu 01:59:25:349.453 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:59:25:349.461 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:59:25:349.469 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: RAC_Connect status: 0 0x1ef81c0 2025-02-06 Thu 01:59:25:349.475 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:59:25:349.483 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:59:25:349.491 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: RAC_Connect status: 0 0x1ef7940 2025-02-06 Thu 01:59:25:349.496 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:59:25:349.504 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.41 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:59:25:349.513 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: RAC_Connect status: 0 0x1ef92c0 2025-02-06 Thu 01:59:25:349.518 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:59:25:349.526 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:59:25:349.534 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: RAC_Connect status: 0 0x1ef9b40 2025-02-06 Thu 01:59:25:349.542 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:59:25:349.550 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:59:25:349.558 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: RAC_Connect status: 0 0x1efa490 2025-02-06 Thu 01:59:25:349.563 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:59:25:349.571 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:59:25:349.579 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: RAC_Connect status: 0 0x1ef8a40 2025-02-06 Thu 01:59:25:349.584 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:59:25:349.592 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:59:25:349.600 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: RAC_Connect status: 0 0x1efade0 2025-02-06 Thu 01:59:25:349.606 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: Connected to REDIS... 2025-02-06 Thu 01:59:25:349.614 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-06 Thu 01:59:25:351.681 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.23:57850::10.0.0.23:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2025-02-06 Thu 01:59:25:351.699 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: 10.0.0.23:57850::10.0.0.23:8086(-1) OnSessionEvent TCP Connected 2025-02-06 Thu 01:59:25:351.707 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: 10.0.0.23:57850::10.0.0.23:8086(-1):EvTcpConnected 2025-02-06 Thu 01:59:25:351.948 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:59:25:351.962 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:59:25:352.005 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:59:25:352.021 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:59:25:352.031 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:59:25:352.044 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:59:25:352.053 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:59:25:354.257 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:59:25:354.278 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:59:25:354.296 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:59:25:354.306 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.23:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:59:25:354.350 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:59:25:354.371 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:59:25:354.387 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:59:25:354.404 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:59:25:354.744 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:59:25:354.762 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:59:25:354.787 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:59:25:355.111 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:59:25:355.127 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:59:25:355.136 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:59:25:355.148 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:59:25:355.168 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:59:25:355.180 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:59:25:355.196 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.41:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:59:25:355.209 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:59:25:355.218 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:59:25:355.231 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:59:25:355.241 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:59:25:355.253 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:59:25:355.263 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:59:25:355.275 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:59:25:355.284 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:59:25:355.297 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-06 Thu 01:59:25:355.306 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-06 Thu 01:59:25:355.329 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:59:25:355.359 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:59:25:355.376 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:59:25:355.391 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:59:25:355.406 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:59:25:355.427 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:59:25:355.452 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:59:25:355.470 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:59:25:355.483 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:59:25:355.497 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-06 Thu 01:59:25:355.583 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-06 Thu 01:59:25:355.617 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: SANDESH: Send FAILED: 1738807165349188 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-06 Thu 01:59:25:355.627 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-06 Thu 01:59:25:355.638 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-06 Thu 01:59:25:355.646 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-06 Thu 01:59:25:355.668 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: ClientInit 2025-02-06 Thu 01:59:25:355.688 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2696-1.:contrail-query-engine:0:Database count 1 2025-02-06 Thu 01:59:25:355.779 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:59:25:355.789 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:59:25:355.812 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: SANDESH: Send FAILED: 1738807165351995 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-06 Thu 01:59:25:355.832 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:59:25:355.842 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:59:25:355.849 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:59:25:355.868 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: SANDESH: Send FAILED: 1738807165354337 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.41:6379::Down.10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:59:25:355.875 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:59:25:355.883 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:59:25:355.889 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:59:25:355.907 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: SANDESH: Send FAILED: 1738807165355350 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = 10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:59:25:355.913 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:59:25:355.920 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:59:25:355.927 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:59:25:355.944 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: SANDESH: Send FAILED: 1738807165355446 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:59:25:355.955 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:59:25:355.963 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:59:25:355.970 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-06 Thu 01:59:25:355.987 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: SANDESH: Send FAILED: 1738807165355662 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-06 Thu 01:59:25:355.993 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-06 Thu 01:59:25:356.001 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:59:25:356.007 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-06 Thu 01:59:25:356.028 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-06 Thu 01:59:25:356.034 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-06 Thu 01:59:25:356.167 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: SANDESH: Send FAILED: 1738807165355768 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738807165302449 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 63, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-06 Thu 01:59:25:356.182 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635569108736, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-06 Thu 01:59:25:361.447 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635526272768, Pid 1]: OnMessage control in state: ClientInit 2025-02-06 Thu 01:59:25:361.486 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635526272768, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-06 Thu 01:59:25:361.496 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635526272768, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-06 Thu 01:59:25:361.521 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635526272768, Pid 1]: Received Ctrl Message with size 0 2025-02-06 Thu 01:59:25:361.656 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635526272768, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1738807165302449 collector_name = collector_ip = 10.0.0.23:8086 collector_list= [ [ (*_iter40) = 10.0.0.23:8086, (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.41:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 63, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2025-02-06 Thu 01:59:25:361.682 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635526272768, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 01:59:25:361.708 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635526272768, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2696-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.23:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.23:9042, (*_iter6) = 10.0.0.41:9042, (*_iter6) = 10.0.0.38:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.23:6379, (*_iter6) = 10.0.0.41:6379, (*_iter6) = 10.0.0.38:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2025-02-06 Thu 01:59:25:361.720 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635526272768, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-06 Thu 01:59:25:361.728 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635526272768, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-06 Thu 01:59:25:361.735 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635526272768, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-06 Thu 01:59:25:361.763 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635526272768, Pid 1]: Established 2025-02-06 Thu 02:00:25:303.533 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139634604488448, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-06 Thu 02:01:25:304.158 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139634608686848, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-06 Thu 02:02:25:304.706 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-06 Thu 02:03:25:305.363 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139634604488448, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-06 Thu 02:04:25:306.092 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139634621282048, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-06 Thu 02:05:25:306.776 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635517875968, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-06 Thu 02:06:25:308.379 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139634617083648, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-06 Thu 02:07:25:309.262 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635501082368, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-06 Thu 02:08:25:309.814 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-06 Thu 02:09:25:313.218 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139634608686848, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-02-06 Thu 02:10:25:314.517 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635513677568, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-02-06 Thu 02:11:25:315.070 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-02-06 Thu 02:12:25:315.665 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635501082368, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-02-06 Thu 02:13:25:316.897 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635517875968, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-02-06 Thu 02:14:07:260.011 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.23:8091::10.0.0.23:57440 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-06 Thu 02:14:25:317.432 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139634608686848, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-02-06 Thu 02:14:31:089.551 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.23:8091::10.0.0.23:57382 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-06 Thu 02:14:46:313.593 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.23:8091::10.0.0.23:34314 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-06 Thu 02:15:00:616.834 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.23:8091::10.0.0.23:39464 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-06 Thu 02:15:25:317.975 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-02-06 Thu 02:15:29:563.157 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.23:8091::10.0.0.23:39336 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-06 Thu 02:16:25:318.492 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635501082368, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-02-06 Thu 02:16:54:373.400 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.23:8091::10.0.0.23:55532 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-06 Thu 02:17:10:650.829 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.23 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 773 2025-02-06 Thu 02:17:10:651.032 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID 78b54a78-e430-11ef-8e60-00000a000017 src/contrail-analytics/contrail-query-engine/query.cc 1166 2025-02-06 Thu 02:17:10:651.069 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 :0:json query is: { end_time: 1738808230598609 , enqueue_time: 1738808230650461 , query_metadata: {"enqueue_time": 1738808230650354} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1738808218387983 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-06 Thu 02:17:10:651.258 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: Chunks: 2 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 880 2025-02-06 Thu 02:17:10:651.275 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 931 2025-02-06 Thu 02:17:10:651.342 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for 78b54a78-e430-11ef-8e60-00000a000017 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 940 2025-02-06 Thu 02:17:10:651.424 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139634612885248, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 :0:json query is: { end_time: 1738808230598609 , enqueue_time: 1738808230650461 , query_metadata: {"enqueue_time": 1738808230650354} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1738808218387983 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-06 Thu 02:17:10:756.118 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QELog: 78b54a78-e430-11ef-8e60-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:10:756.156 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QELog: 78b54a78-e430-11ef-8e60-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:10:756.167 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QELog: 78b54a78-e430-11ef-8e60-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:10:756.176 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QELog: 78b54a78-e430-11ef-8e60-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:10:756.185 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QELog: 78b54a78-e430-11ef-8e60-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:10:756.201 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QELog: 78b54a78-e430-11ef-8e60-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:10:756.210 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QELog: 78b54a78-e430-11ef-8e60-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:10:756.218 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QELog: 78b54a78-e430-11ef-8e60-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:10:756.226 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QELog: 78b54a78-e430-11ef-8e60-00000a000017 UNION between tables of sizes 0 and 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:10:756.238 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QELog: 78b54a78-e430-11ef-8e60-00000a000017 Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:10:756.530 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:756.697 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:756.867 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:757.019 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:757.168 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:757.354 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QELog: 78b54a78-e430-11ef-8e60-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:10:757.365 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QELog: 78b54a78-e430-11ef-8e60-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:10:757.374 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QELog: 78b54a78-e430-11ef-8e60-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:10:757.387 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QELog: 78b54a78-e430-11ef-8e60-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:10:757.395 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QELog: 78b54a78-e430-11ef-8e60-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:10:757.402 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QELog: 78b54a78-e430-11ef-8e60-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:10:757.410 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QELog: 78b54a78-e430-11ef-8e60-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:10:757.418 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QELog: 78b54a78-e430-11ef-8e60-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:10:757.425 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QELog: 78b54a78-e430-11ef-8e60-00000a000017 UNION between tables of sizes 0 and 39 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:10:757.477 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QELog: 78b54a78-e430-11ef-8e60-00000a000017 Resulting size of set 39 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:10:757.535 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 :0:json query is: { end_time: 1738808230598609 , enqueue_time: 1738808230650461 , query_metadata: {"enqueue_time": 1738808230650354} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1738808218387983 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-06 Thu 02:17:10:757.738 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:757.907 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:758.065 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:758.219 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:758.367 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:758.517 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:758.664 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:758.820 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:758.968 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:759.120 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:759.268 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:759.415 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:759.561 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:759.710 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:759.869 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:760.017 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:760.176 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:760.325 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:760.473 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:760.619 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:760.774 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:760.921 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:761.067 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:761.216 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:761.365 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:761.512 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:761.657 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:761.812 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:761.960 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:764.007 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:764.187 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:764.338 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:764.486 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:764.640 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:764.797 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:764.953 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:765.100 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:765.245 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:765.393 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: [SYS_INFO]: QEQueryLog: 78b54a78-e430-11ef-8e60-00000a000017 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:10:765.850 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635513677568, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 44 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 520 2025-02-06 Thu 02:17:10:766.370 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635513677568, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 44 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 528 2025-02-06 Thu 02:17:10:766.579 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635513677568, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":6, "count":44} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 563 2025-02-06 Thu 02:17:10:766.905 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635513677568, Pid 1]: [SYS_INFO]: QELog: Finished: QID 78b54a78-e430-11ef-8e60-00000a000017 Table ObjectVNTable Time(ms) 115 RedisTime(ms) 0 MergeTime(ms) 0 Rows 44 EnQ-delay0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 644 2025-02-06 Thu 02:17:10:766.947 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635564910336, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-2696-1. table = ObjectVNTable query_stats= [ qid = 78b54a78-e430-11ef-8e60-00000a000017 where = [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-27815585:ctest-vn22-78729941", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] } post = time_span = 8 error = None chunks = 2 chunk_where_time = 102,0, 105,0, chunk_select_time = 0,0, 0,7, chunk_postproc_time = 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 115 rows = 44 enq_delay = 0 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 636 2025-02-06 Thu 02:17:10:766.998 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635513677568, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 676 2025-02-06 Thu 02:17:13:764.935 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.23 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 773 2025-02-06 Thu 02:17:13:765.163 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID 7a9080e2-e430-11ef-9107-00000a000017 src/contrail-analytics/contrail-query-engine/query.cc 1166 2025-02-06 Thu 02:17:13:765.198 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QEQueryLog: 7a9080e2-e430-11ef-9107-00000a000017 :0:json query is: { end_time: 1738808233752836 , enqueue_time: 1738808233764578 , query_metadata: {"enqueue_time": 1738808233764473} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1738808218387983 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "e40a5c59-fd31-4795-9e7d-51708d756aa6", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-06 Thu 02:17:13:765.364 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: Chunks: 2 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 880 2025-02-06 Thu 02:17:13:765.381 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 931 2025-02-06 Thu 02:17:13:765.446 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for 7a9080e2-e430-11ef-9107-00000a000017 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 940 2025-02-06 Thu 02:17:13:765.649 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139634621282048, Pid 1]: [SYS_INFO]: QEQueryLog: 7a9080e2-e430-11ef-9107-00000a000017 :0:json query is: { end_time: 1738808233752836 , enqueue_time: 1738808233764578 , query_metadata: {"enqueue_time": 1738808233764473} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1738808218387983 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "e40a5c59-fd31-4795-9e7d-51708d756aa6", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-06 Thu 02:17:13:801.636 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635547256576, Pid 1]: [SYS_INFO]: QELog: 7a9080e2-e430-11ef-9107-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:13:801.674 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635547256576, Pid 1]: [SYS_INFO]: QELog: 7a9080e2-e430-11ef-9107-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:13:801.685 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635547256576, Pid 1]: [SYS_INFO]: QELog: 7a9080e2-e430-11ef-9107-00000a000017 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:13:801.702 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635501082368, Pid 1]: [SYS_INFO]: QELog: 7a9080e2-e430-11ef-9107-00000a000017 UNION between tables of sizes 0 and 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:13:801.705 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635547256576, Pid 1]: [SYS_INFO]: QELog: 7a9080e2-e430-11ef-9107-00000a000017 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:13:801.762 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635501082368, Pid 1]: [SYS_INFO]: QELog: 7a9080e2-e430-11ef-9107-00000a000017 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:13:801.763 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635547256576, Pid 1]: [SYS_INFO]: QELog: 7a9080e2-e430-11ef-9107-00000a000017 UNION between tables of sizes 0 and 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:13:801.777 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635501082368, Pid 1]: [SYS_INFO]: QELog: 7a9080e2-e430-11ef-9107-00000a000017 UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:13:801.799 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635547256576, Pid 1]: [SYS_INFO]: QELog: 7a9080e2-e430-11ef-9107-00000a000017 Resulting size of set 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:13:801.802 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635501082368, Pid 1]: [SYS_INFO]: QELog: 7a9080e2-e430-11ef-9107-00000a000017 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:13:801.817 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635547256576, Pid 1]: [SYS_INFO]: QELog: 7a9080e2-e430-11ef-9107-00000a000017 UNION between tables of sizes 3 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:13:801.825 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635501082368, Pid 1]: [SYS_INFO]: QELog: 7a9080e2-e430-11ef-9107-00000a000017 UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:13:801.839 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635547256576, Pid 1]: [SYS_INFO]: QELog: 7a9080e2-e430-11ef-9107-00000a000017 Resulting size of set 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:13:801.845 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635501082368, Pid 1]: [SYS_INFO]: QELog: 7a9080e2-e430-11ef-9107-00000a000017 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:13:801.857 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635547256576, Pid 1]: [SYS_INFO]: QELog: 7a9080e2-e430-11ef-9107-00000a000017 UNION between tables of sizes 3 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:13:801.865 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635501082368, Pid 1]: [SYS_INFO]: QELog: 7a9080e2-e430-11ef-9107-00000a000017 UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:13:801.878 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635547256576, Pid 1]: [SYS_INFO]: QELog: 7a9080e2-e430-11ef-9107-00000a000017 Resulting size of set 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:13:801.891 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635501082368, Pid 1]: [SYS_INFO]: QELog: 7a9080e2-e430-11ef-9107-00000a000017 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:13:801.918 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635501082368, Pid 1]: [SYS_INFO]: QELog: 7a9080e2-e430-11ef-9107-00000a000017 UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-06 Thu 02:17:13:801.927 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635501082368, Pid 1]: [SYS_INFO]: QELog: 7a9080e2-e430-11ef-9107-00000a000017 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-06 Thu 02:17:13:801.949 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635547256576, Pid 1]: [SYS_INFO]: QEQueryLog: 7a9080e2-e430-11ef-9107-00000a000017 :0:json query is: { end_time: 1738808233752836 , enqueue_time: 1738808233764578 , query_metadata: {"enqueue_time": 1738808233764473} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1738808218387983 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "e40a5c59-fd31-4795-9e7d-51708d756aa6", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-06 Thu 02:17:13:802.195 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635547256576, Pid 1]: [SYS_INFO]: QEQueryLog: 7a9080e2-e430-11ef-9107-00000a000017 ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id e40a5c59-fd31-4795-9e7d-51708d756aa6 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:13:802.221 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635501082368, Pid 1]: [SYS_INFO]: QEQueryLog: 7a9080e2-e430-11ef-9107-00000a000017 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id e40a5c59-fd31-4795-9e7d-51708d756aa6 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:13:802.405 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635547256576, Pid 1]: [SYS_INFO]: QEQueryLog: 7a9080e2-e430-11ef-9107-00000a000017 ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id e40a5c59-fd31-4795-9e7d-51708d756aa6 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:13:802.431 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635501082368, Pid 1]: [SYS_INFO]: QEQueryLog: 7a9080e2-e430-11ef-9107-00000a000017 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id e40a5c59-fd31-4795-9e7d-51708d756aa6 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:13:802.578 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635547256576, Pid 1]: [SYS_INFO]: QEQueryLog: 7a9080e2-e430-11ef-9107-00000a000017 ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id e40a5c59-fd31-4795-9e7d-51708d756aa6 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-06 Thu 02:17:13:802.759 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635501082368, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 520 2025-02-06 Thu 02:17:13:802.836 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635501082368, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 528 2025-02-06 Thu 02:17:13:802.944 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635501082368, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":5} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 563 2025-02-06 Thu 02:17:13:803.201 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635526272768, Pid 1]: [SYS_INFO]: QELog: Finished: QID 7a9080e2-e430-11ef-9107-00000a000017 Table ObjectVMTable Time(ms) 38 RedisTime(ms) 0 MergeTime(ms) 0 Rows 5 EnQ-delay0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 644 2025-02-06 Thu 02:17:13:803.243 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635526272768, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 676 2025-02-06 Thu 02:17:13:803.312 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635526272768, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-2696-1. table = ObjectVMTable query_stats= [ qid = 7a9080e2-e430-11ef-9107-00000a000017 where = [[{"name": "ObjectId", "value": "e40a5c59-fd31-4795-9e7d-51708d756aa6", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 2 chunk_where_time = 36,0, 36,0, chunk_select_time = 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 38 rows = 5 enq_delay = 0 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 636 2025-02-06 Thu 02:17:25:319.210 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635522074368, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-02-06 Thu 02:18:25:319.897 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139634604488448, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-02-06 Thu 02:19:25:320.550 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139634621282048, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-02-06 Thu 02:20:25:321.079 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635522074368, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-02-06 Thu 02:21:25:322.355 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635509479168, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-02-06 Thu 02:22:25:322.914 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139634608686848, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-02-06 Thu 02:23:25:324.126 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635522074368, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-02-06 Thu 02:23:32:450.854 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.23:8091::10.0.0.23:39876 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-06 Thu 02:24:25:324.656 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635505280768, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-02-06 Thu 02:25:08:617.415 UTC cn-jenkins-deploy-platform-ansible-os-2696-1 [Thread 139635716970624, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.23:8091::10.0.0.23:37888 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256