2025-02-13 Thu 00:50:26:640.395 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: http-server-port 8091 2025-02-13 Thu 00:50:26:640.495 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Max-tasks 16 2025-02-13 Thu 00:50:26:640.505 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Max-slice 100 2025-02-13 Thu 00:50:26:640.511 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Collectors 10.0.0.46:8086 2025-02-13 Thu 00:50:26:640.517 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Collectors 10.0.0.35:8086 2025-02-13 Thu 00:50:26:640.523 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Collectors 10.0.0.12:8086 2025-02-13 Thu 00:50:26:640.544 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: SANDESH: ROLE : Generator 2025-02-13 Thu 00:50:26:640.550 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-13 Thu 00:50:26:640.557 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2728-1. 2025-02-13 Thu 00:50:26:640.562 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-13 Thu 00:50:26:640.568 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-13 Thu 00:50:26:640.574 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-13 Thu 00:50:26:640.581 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-13 Thu 00:50:26:641.264 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: HTTP Introspect Init 2025-02-13 Thu 00:50:26:641.394 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-13 Thu 00:50:26:641.460 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Sandesh Http Server Port: 8091 2025-02-13 Thu 00:50:26:641.514 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-13 Thu 00:50:26:641.528 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-13 Thu 00:50:26:641.639 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Idle 2025-02-13 Thu 00:50:26:641.823 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: SANDESH: No Client: 1739407826641738 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 00:50:26:641.999 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-13 Thu 00:50:26:642.018 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-13 Thu 00:50:26:642.032 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Cassandra Servers: 10.0.0.46:9042 10.0.0.35:9042 10.0.0.12:9042 2025-02-13 Thu 00:50:26:644.010 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Processing scm::EvStart in state Idle 2025-02-13 Thu 00:50:26:644.085 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Disconnect 2025-02-13 Thu 00:50:26:644.165 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-13 Thu 00:50:26:644.442 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Connect : Start Connect timer 10.0.0.46:8086 2025-02-13 Thu 00:50:26:644.570 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 00:50:26:644.597 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: SANDESH: Send FAILED: 1739407826644075 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-13 Thu 00:50:26:644.608 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 00:50:26:644.619 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 00:50:26:644.709 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: SANDESH: Send FAILED: 1739407826644153 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 00:50:26:644.729 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 00:50:26:644.740 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 00:50:26:644.819 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: SANDESH: Send FAILED: 1739407826644220 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 00:50:26:644.834 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 00:50:26:644.845 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 00:50:26:644.862 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: SANDESH: Send FAILED: 1739407826644243 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-13 Thu 00:50:26:644.869 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 00:50:26:644.878 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 00:50:26:644.951 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: SANDESH: Send FAILED: 1739407826644550 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 00:50:26:644.965 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 00:50:26:650.763 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-13 Thu 00:50:26:650.795 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:50:26:650.859 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:50:26:650.894 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300752185088, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 00:50:26:650.927 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300752185088, Pid 1]: SANDESH: Send FAILED: 1739407826650834 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-13 Thu 00:50:26:650.937 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300752185088, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 00:50:31:662.642 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-13 Thu 00:50:31:662.696 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:50:31:662.737 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:50:36:669.524 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-13 Thu 00:50:36:669.571 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:50:36:669.596 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:50:41:678.426 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-13 Thu 00:50:41:678.472 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:50:41:678.497 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:50:46:857.612 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-13 Thu 00:50:46:857.660 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:50:46:857.687 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:50:51:910.780 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-13 Thu 00:50:51:910.833 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:50:51:910.859 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:50:56:948.825 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-13 Thu 00:50:56:948.881 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:50:56:948.906 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:51:01:991.884 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-13 Thu 00:51:01:991.941 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:51:01:991.966 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 7 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:51:07:049.536 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-13 Thu 00:51:07:049.604 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:51:07:049.633 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 8 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:51:12:099.916 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-13 Thu 00:51:12:099.961 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:51:12:099.988 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 9 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:51:17:191.904 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-13 Thu 00:51:17:191.956 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:51:17:191.982 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 10 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:51:22:254.885 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-13 Thu 00:51:22:254.932 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:51:22:254.956 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 11 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:51:27:286.942 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-13 Thu 00:51:27:287.008 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 12 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-13 Thu 00:51:32:323.626 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-13 Thu 00:51:37:329.545 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 00:51:37:329.592 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-13 Thu 00:51:37:329.621 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: SANDESH: Send FAILED: 1739407897329449 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-13 Thu 00:51:37:329.638 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 00:51:37:329.625 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: -(-1):EvTcpConnectFail 2025-02-13 Thu 00:51:37:330.155 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf4620 2025-02-13 Thu 00:51:37:330.166 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:330.183 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:330.200 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf4d40 2025-02-13 Thu 00:51:37:330.206 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:330.216 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:330.226 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bee070 2025-02-13 Thu 00:51:37:330.232 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:330.241 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:330.252 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf5140 2025-02-13 Thu 00:51:37:330.257 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:330.266 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:330.276 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf58b0 2025-02-13 Thu 00:51:37:330.282 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:330.291 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:330.309 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf6180 2025-02-13 Thu 00:51:37:330.315 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:330.324 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:330.334 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf7400 2025-02-13 Thu 00:51:37:330.340 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:330.348 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:330.358 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf8fc0 2025-02-13 Thu 00:51:37:330.364 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:330.373 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:330.383 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf6ac0 2025-02-13 Thu 00:51:37:330.389 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:330.397 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:330.406 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf7d40 2025-02-13 Thu 00:51:37:330.412 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:330.421 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:330.431 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf8680 2025-02-13 Thu 00:51:37:330.440 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:330.454 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:330.469 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bfab80 2025-02-13 Thu 00:51:37:330.479 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:330.489 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:330.499 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bfa240 2025-02-13 Thu 00:51:37:330.505 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:330.518 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:330.528 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf9900 2025-02-13 Thu 00:51:37:330.533 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:330.543 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:330.552 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bfb4c0 2025-02-13 Thu 00:51:37:330.557 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 00:51:37:330.566 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 00:51:37:330.158 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-13 Thu 00:51:37:330.705 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Connect : EvTcpConnectFail 2025-02-13 Thu 00:51:37:330.811 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:330.828 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:330.856 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-13 Thu 00:51:37:330.885 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:330.903 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Idle 2025-02-13 Thu 00:51:37:330.908 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:330.945 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:330.980 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:331.004 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 00:51:37:331.007 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:331.070 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:331.109 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:331.163 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: SANDESH: Send FAILED: 1739407897330797 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 54 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 55, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 00:51:37:331.193 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 00:51:37:331.209 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 00:51:37:331.178 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:331.241 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: SANDESH: Send FAILED: 1739407897330829 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-13 Thu 00:51:37:331.278 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 00:51:37:331.289 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 00:51:37:331.295 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:331.313 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: SANDESH: Send FAILED: 1739407897330874 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 00:51:37:331.319 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:331.338 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 00:51:37:331.350 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 00:51:37:331.353 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:331.374 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:331.398 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:331.400 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 00:51:37:331.418 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:331.455 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:331.471 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:331.489 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:331.503 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:331.536 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:331.571 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:331.577 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: SANDESH: Send FAILED: 1739407897330989 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 60 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 62, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 00:51:37:331.597 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 00:51:37:331.603 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:331.627 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:331.643 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:331.654 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:331.668 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:331.683 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:331.608 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 00:51:37:331.731 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: SANDESH: Send FAILED: 1739407897331169 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.35:6379::Down.10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 00:51:37:331.697 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:331.747 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 00:51:37:331.769 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:331.783 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 00:51:37:331.803 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:331.816 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: SANDESH: Send FAILED: 1739407897331594 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 00:51:37:331.816 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:331.854 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 00:51:37:331.871 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:331.893 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:331.914 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:331.931 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:331.946 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 00:51:37:331.956 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 00:51:37:331.977 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:332.223 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:332.285 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:332.312 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:332.320 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 00:51:37:332.329 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:332.384 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:332.421 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 00:51:37:332.389 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: SANDESH: Send FAILED: 1739407897331895 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 17 dequeues = 17 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739407897330901 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739407897331781 ] max_sm_queue_count = 5 ] 2025-02-13 Thu 00:51:37:332.648 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 00:51:37:332.667 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 00:51:37:332.690 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: SANDESH: Send FAILED: 1739407897331999 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 00:51:37:332.698 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 00:51:37:332.708 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 00:51:37:332.839 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: SANDESH: Send FAILED: 1739407897332297 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 94 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 96, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 13, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 00:51:37:332.856 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 00:51:38:331.048 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-13 Thu 00:51:38:331.306 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: Connect : Start Connect timer 10.0.0.35:8086 2025-02-13 Thu 00:51:38:331.423 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 00:51:38:331.456 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: SANDESH: Send FAILED: 1739407898331127 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 00:51:38:331.466 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 00:51:38:331.477 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 00:51:38:333.843 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.46:34970::10.0.0.35:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-13 Thu 00:51:38:333.872 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:34970::10.0.0.35:8086(-1) OnSessionEvent TCP Connected 2025-02-13 Thu 00:51:38:333.881 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:34970::10.0.0.35:8086(-1):EvTcpConnected 2025-02-13 Thu 00:51:38:334.128 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: SANDESH: Send FAILED: 1739407898331411 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 99 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 101, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 00:51:38:334.155 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 00:51:38:334.171 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-13 Thu 00:51:38:334.181 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-13 Thu 00:51:38:334.215 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: ClientInit 2025-02-13 Thu 00:51:38:334.250 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2728-1.:contrail-query-engine:0:Database count 1 2025-02-13 Thu 00:51:38:334.333 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 00:51:38:334.343 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 00:51:38:334.368 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: SANDESH: Send FAILED: 1739407898334208 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 00:51:38:334.376 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 00:51:38:334.386 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 00:51:38:334.393 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-13 Thu 00:51:38:334.406 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 00:51:38:334.418 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-13 Thu 00:51:38:334.539 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: SANDESH: Send FAILED: 1739407898334323 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 99 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 102, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 00:51:38:334.553 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-13 Thu 00:51:38:336.977 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: OnMessage control in state: ClientInit 2025-02-13 Thu 00:51:38:337.059 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-13 Thu 00:51:38:337.071 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-13 Thu 00:51:38:337.099 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: Received Ctrl Message with size 0 2025-02-13 Thu 00:51:38:337.270 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 17 dequeues = 17 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739407897330901 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739407897331781 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 99 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 102, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 5 proxy seq 10 2025-02-13 Thu 00:51:38:337.296 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-13 Thu 00:51:38:337.323 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2025-02-13 Thu 00:51:38:337.340 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-13 Thu 00:51:38:337.349 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-13 Thu 00:51:38:337.358 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-13 Thu 00:51:38:337.387 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: Established 2025-02-13 Thu 00:51:48:500.251 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: OnMessage control in state: Established 2025-02-13 Thu 00:51:48:500.332 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2025-02-13 Thu 00:51:48:500.344 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: Established : EvSandeshMessageRecv 2025-02-13 Thu 00:51:48:500.392 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: Received Ctrl Message with size 3 2025-02-13 Thu 00:51:48:500.407 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 11, total = 0 2025-02-13 Thu 00:51:48:500.420 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 13, total = 0 2025-02-13 Thu 00:51:48:500.433 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2025-02-13 Thu 00:51:48:500.442 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-13 Thu 00:52:07:332.574 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-13 Thu 00:53:07:333.244 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300634789632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-13 Thu 00:54:07:333.830 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-13 Thu 00:54:45:058.306 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.46:8091::10.0.0.46:33408 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-02-13 Thu 00:55:07:334.659 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-13 Thu 00:55:08:809.300 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.46:8091::10.0.0.46:57890 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-02-13 Thu 00:56:07:335.573 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-13 Thu 00:57:07:336.465 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-13 Thu 00:58:07:337.527 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-13 Thu 00:59:07:338.166 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-13 Thu 01:00:07:338.821 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-13 Thu 01:01:07:340.095 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-02-13 Thu 01:02:07:340.839 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-02-13 Thu 01:03:07:341.424 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-02-13 Thu 01:04:07:341.993 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-02-13 Thu 01:05:07:342.577 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-02-13 Thu 01:06:07:343.549 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-02-13 Thu 01:07:07:344.159 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-02-13 Thu 01:08:07:345.040 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-02-13 Thu 01:09:07:345.576 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-02-13 Thu 01:10:07:346.435 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300064380672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-02-13 Thu 01:11:07:347.234 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-02-13 Thu 01:12:07:347.903 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-02-13 Thu 01:13:07:348.612 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300634789632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-02-13 Thu 01:14:07:349.166 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300634789632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-02-13 Thu 01:15:07:349.733 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-02-13 Thu 01:16:07:350.354 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-02-13 Thu 01:17:07:350.931 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2025-02-13 Thu 01:18:07:351.679 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300064380672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2025-02-13 Thu 01:19:07:352.831 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2025-02-13 Thu 01:20:07:353.474 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300064380672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0 2025-02-13 Thu 01:21:07:354.989 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 30 without seqno , total = 0 2025-02-13 Thu 01:22:07:355.619 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 31 without seqno , total = 0 2025-02-13 Thu 01:23:07:356.234 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 32 without seqno , total = 0 2025-02-13 Thu 01:24:07:356.835 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 33 without seqno , total = 0 2025-02-13 Thu 01:25:07:357.711 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 34 without seqno , total = 0 2025-02-13 Thu 01:26:07:358.540 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 35 without seqno , total = 0 2025-02-13 Thu 01:27:07:359.577 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300064380672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 36 without seqno , total = 0 2025-02-13 Thu 01:28:07:360.292 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 37 without seqno , total = 0 2025-02-13 Thu 01:29:07:361.140 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 38 without seqno , total = 0 2025-02-13 Thu 01:30:07:362.216 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 39 without seqno , total = 0 2025-02-13 Thu 01:31:07:362.835 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 40 without seqno , total = 0 2025-02-13 Thu 01:32:07:363.866 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300064380672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 41 without seqno , total = 0 2025-02-13 Thu 01:33:07:364.480 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 42 without seqno , total = 0 2025-02-13 Thu 01:34:07:365.088 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 43 without seqno , total = 0 2025-02-13 Thu 01:35:07:365.652 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 44 without seqno , total = 0 2025-02-13 Thu 01:36:07:366.228 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 45 without seqno , total = 0 2025-02-13 Thu 01:37:07:366.899 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 46 without seqno , total = 0 2025-02-13 Thu 01:38:07:368.145 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300064380672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 47 without seqno , total = 0 2025-02-13 Thu 01:39:07:368.886 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 48 without seqno , total = 0 2025-02-13 Thu 01:40:07:369.500 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300064380672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 49 without seqno , total = 0 2025-02-13 Thu 01:41:07:370.200 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 50 without seqno , total = 0 2025-02-13 Thu 01:42:07:370.875 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 51 without seqno , total = 0 2025-02-13 Thu 01:43:07:371.544 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300651575040, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 52 without seqno , total = 0 2025-02-13 Thu 01:44:07:372.127 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 53 without seqno , total = 0 2025-02-13 Thu 01:45:07:372.841 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 54 without seqno , total = 0 2025-02-13 Thu 01:46:07:373.519 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300064380672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 55 without seqno , total = 0 2025-02-13 Thu 01:47:07:374.103 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 56 without seqno , total = 0 2025-02-13 Thu 01:48:07:374.725 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300064380672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 57 without seqno , total = 0 2025-02-13 Thu 01:49:07:375.355 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 58 without seqno , total = 0 2025-02-13 Thu 01:50:07:376.004 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 59 without seqno , total = 0 2025-02-13 Thu 01:51:07:376.682 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 60 without seqno , total = 0 2025-02-13 Thu 01:52:07:377.362 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300651575040, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 61 without seqno , total = 0 2025-02-13 Thu 01:53:07:377.938 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 62 without seqno , total = 0 2025-02-13 Thu 01:54:07:378.566 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 63 without seqno , total = 0 2025-02-13 Thu 01:55:07:379.212 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 64 without seqno , total = 0 2025-02-13 Thu 01:56:07:379.764 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 65 without seqno , total = 0 2025-02-13 Thu 01:57:07:380.401 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 66 without seqno , total = 0 2025-02-13 Thu 01:58:07:381.024 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 67 without seqno , total = 0 2025-02-13 Thu 01:59:07:381.710 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 68 without seqno , total = 0 2025-02-13 Thu 02:00:07:382.340 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 69 without seqno , total = 0 2025-02-13 Thu 02:01:07:383.178 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 70 without seqno , total = 0 2025-02-13 Thu 02:02:07:384.674 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 71 without seqno , total = 0 2025-02-13 Thu 02:03:07:385.266 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 72 without seqno , total = 0 2025-02-13 Thu 02:04:07:385.866 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 73 without seqno , total = 0 2025-02-13 Thu 02:05:07:387.147 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 74 without seqno , total = 0 2025-02-13 Thu 02:06:07:387.750 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300651575040, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 75 without seqno , total = 0 2025-02-13 Thu 02:07:07:388.472 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300676757248, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 76 without seqno , total = 0 2025-02-13 Thu 02:07:39:930.137 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bf9900 2025-02-13 Thu 02:07:39:930.202 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:07:39:930.251 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bfab80 2025-02-13 Thu 02:07:39:930.265 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:07:39:930.290 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bfa240 2025-02-13 Thu 02:07:39:930.301 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:07:39:930.338 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-13 Thu 02:07:39:930.346 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bf8fc0 2025-02-13 Thu 02:07:39:930.356 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:07:39:930.382 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bfb4c0 2025-02-13 Thu 02:07:39:930.393 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:07:39:930.808 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:07:39:930.856 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:07:39:930.872 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:07:39:930.961 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:07:39:930.976 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:07:39:998.580 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bf8680 2025-02-13 Thu 02:07:39:998.647 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:07:39:998.785 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bf7d40 2025-02-13 Thu 02:07:39:998.801 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:07:39:998.828 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bf6ac0 2025-02-13 Thu 02:07:39:998.839 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:07:39:998.860 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bf7400 2025-02-13 Thu 02:07:39:998.870 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:07:39:998.896 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-13 Thu 02:07:39:998.904 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bf6180 2025-02-13 Thu 02:07:39:998.914 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:07:39:999.138 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:07:39:999.363 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:07:39:999.384 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:07:39:999.403 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:07:39:999.419 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:07:44:930.907 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:07:44:931.109 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:07:44:931.173 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:07:44:931.235 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:07:44:931.284 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:07:44:931.494 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:07:44:931.540 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:07:44:931.579 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:07:44:931.598 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:07:44:931.615 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:07:44:999.242 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:07:44:999.433 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:07:44:999.475 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:07:44:999.503 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:07:44:999.537 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:07:44:999.933 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:07:44:999.972 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:07:44:999.992 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:07:45:000.008 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:07:45:000.030 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:07:49:931.579 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:07:49:931.769 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:07:49:931.817 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:07:49:931.869 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:07:49:931.899 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:07:49:932.150 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:07:49:932.183 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:07:49:932.198 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:07:49:932.224 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:07:49:932.239 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:07:49:996.459 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.46:34970::10.0.0.35:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2025-02-13 Thu 02:07:49:996.511 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.46:34970::10.0.0.35:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-13 Thu 02:07:49:996.538 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:34970::10.0.0.35:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-13 Thu 02:07:49:996.547 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:34970::10.0.0.35:8086(-1):EvTcpClose 2025-02-13 Thu 02:07:49:996.625 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvTcpClose in state Established 2025-02-13 Thu 02:07:49:996.638 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Established : EvTcpClose 2025-02-13 Thu 02:07:49:996.921 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-02-13 Thu 02:07:49:997.021 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:07:49:997.199 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SANDESH: Send FAILED: 1739412469996740 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = cn-jenkins-deploy-platform-ansible-os-2728-2. collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3696659 calls = 388 average_bytes = 9527 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 77 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 111 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 306 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 118, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 389, ] ] ] 2025-02-13 Thu 02:07:49:997.227 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:07:49:997.239 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:07:49:997.265 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SANDESH: Send FAILED: 1739412469996776 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.35:6379::Down.10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:07:49:997.273 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:07:49:997.284 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2025-02-13 Thu 02:07:49:997.331 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:07:49:997.484 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SANDESH: Send FAILED: 1739412469997005 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3696659 calls = 388 average_bytes = 9527 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 77 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 111 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 306 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 118, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 389, ] ] ] 2025-02-13 Thu 02:07:49:997.503 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:07:49:999.807 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.46:59844::10.0.0.12:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-13 Thu 02:07:50:010.655 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:59844::10.0.0.12:8086(-1) OnSessionEvent TCP Connected 2025-02-13 Thu 02:07:50:010.681 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:59844::10.0.0.12:8086(-1):EvTcpConnected 2025-02-13 Thu 02:07:50:011.738 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-13 Thu 02:07:50:011.757 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-13 Thu 02:07:50:011.809 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: ClientInit 2025-02-13 Thu 02:07:50:011.866 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2728-1.:contrail-query-engine:0:Database count 2 2025-02-13 Thu 02:07:50:011.976 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:07:50:011.988 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:07:50:012.017 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: SANDESH: Send FAILED: 1739412470011798 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.35:6379::Down.10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:07:50:012.026 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:07:50:012.036 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:07:50:012.044 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-13 Thu 02:07:50:012.057 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:07:50:012.075 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-13 Thu 02:07:50:012.230 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: SANDESH: Send FAILED: 1739412470011964 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3696659 calls = 388 average_bytes = 9527 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 77 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 111 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 306 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 119, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 389, ] ] ] 2025-02-13 Thu 02:07:50:012.247 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-13 Thu 02:07:50:012.956 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:07:50:013.045 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:07:50:013.079 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:07:50:013.112 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:07:50:013.147 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:07:50:013.587 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:07:50:013.624 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:07:50:013.641 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:07:50:013.666 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:07:50:013.682 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:07:50:076.191 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.46:59844::10.0.0.12:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2025-02-13 Thu 02:07:50:076.243 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.46:59844::10.0.0.12:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-13 Thu 02:07:50:076.272 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:59844::10.0.0.12:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-13 Thu 02:07:50:076.282 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:59844::10.0.0.12:8086(-1):EvTcpClose 2025-02-13 Thu 02:07:50:076.770 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-13 Thu 02:07:50:076.787 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: ClientInit : EvTcpClose 2025-02-13 Thu 02:07:50:076.960 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Return to idle with 1000 2025-02-13 Thu 02:07:50:076.971 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Idle 2025-02-13 Thu 02:07:50:077.039 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:07:50:077.207 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: SANDESH: Send FAILED: 1739412470076881 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3698232 calls = 389 average_bytes = 9507 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 77 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 111 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 306 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 121, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 25, _iter59->first = sent _iter59->second = 390, ] ] ] 2025-02-13 Thu 02:07:50:077.233 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:07:50:077.245 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:07:50:077.270 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: SANDESH: Send FAILED: 1739412470076930 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.35:6379::Down.10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:07:50:077.279 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:07:50:077.289 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:07:50:077.299 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:07:50:077.453 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: SANDESH: Send FAILED: 1739412470077027 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3698232 calls = 389 average_bytes = 9507 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 77 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 111 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 306 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 121, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 25, _iter59->first = sent _iter59->second = 390, ] ] ] 2025-02-13 Thu 02:07:50:077.472 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:07:51:077.123 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-13 Thu 02:07:51:077.590 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Connect : Start Connect timer 10.0.0.46:8086 2025-02-13 Thu 02:07:51:077.728 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:07:51:077.764 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: SANDESH: Send FAILED: 1739412471077220 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.35:6379::Down.10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:07:51:077.775 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:07:51:077.787 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:07:51:077.979 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: SANDESH: Send FAILED: 1739412471077706 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3698232 calls = 389 average_bytes = 9507 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 77 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 111 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 306 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 121, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 28, _iter59->first = sent _iter59->second = 390, ] ] ] 2025-02-13 Thu 02:07:51:078.008 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:07:51:080.222 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.46:60282::10.0.0.46:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-13 Thu 02:07:51:080.246 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:60282::10.0.0.46:8086(-1) OnSessionEvent TCP Connected 2025-02-13 Thu 02:07:51:080.255 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:60282::10.0.0.46:8086(-1):EvTcpConnected 2025-02-13 Thu 02:07:51:080.952 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-13 Thu 02:07:51:080.968 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-13 Thu 02:07:51:081.011 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: ClientInit 2025-02-13 Thu 02:07:51:081.029 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2728-1.:contrail-query-engine:0:Database count 3 2025-02-13 Thu 02:07:51:081.147 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:07:51:081.159 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:07:51:081.186 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: SANDESH: Send FAILED: 1739412471081003 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.35:6379::Down.10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:07:51:081.195 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:07:51:081.214 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:07:51:081.222 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-13 Thu 02:07:51:081.235 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:07:51:081.242 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-13 Thu 02:07:51:081.423 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: SANDESH: Send FAILED: 1739412471081134 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3698232 calls = 389 average_bytes = 9507 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 77 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 111 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 306 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 122, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 390, ] ] ] 2025-02-13 Thu 02:07:51:081.441 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300622194432, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-13 Thu 02:07:51:083.426 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: OnMessage control in state: ClientInit 2025-02-13 Thu 02:07:51:083.469 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-13 Thu 02:07:51:083.479 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-13 Thu 02:07:51:083.505 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: Received Ctrl Message with size 0 2025-02-13 Thu 02:07:51:083.737 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3698232 calls = 389 average_bytes = 9507 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 404 dequeues = 404 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 412 dequeues = 412 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Established state_since = 1739407898337386 last_state = Established last_event = scm::EvSandeshSend last_event_at = 1739412427388582 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 384 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 3665613 calls = 383 average_bytes = 9570 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 77 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 111 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 306 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 122, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 390, ] ] max_sm_queue_count = 5 proxy seq 322 2025-02-13 Thu 02:07:51:083.769 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-13 Thu 02:07:51:083.799 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.35:6379::Down.10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 20 2025-02-13 Thu 02:07:51:083.813 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-13 Thu 02:07:51:083.876 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2728-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 4 available_connections = 4 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 77 2025-02-13 Thu 02:07:51:083.889 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-02-13 Thu 02:07:51:083.899 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-13 Thu 02:07:51:083.931 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300072777472, Pid 1]: Established 2025-02-13 Thu 02:07:54:932.241 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:07:54:932.397 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:07:54:932.447 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:07:54:932.482 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:07:54:932.513 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:07:54:932.707 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:07:54:932.749 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:07:54:932.782 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:07:54:932.797 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:07:54:932.812 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:07:55:013.673 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:07:55:013.824 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:07:55:013.872 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:07:55:013.905 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:07:55:013.952 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:07:55:014.263 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf8680 2025-02-13 Thu 02:07:55:014.274 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:07:55:014.329 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:07:55:014.359 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf6180 2025-02-13 Thu 02:07:55:014.366 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:07:55:014.375 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:07:55:014.385 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf7d40 2025-02-13 Thu 02:07:55:014.390 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:07:55:014.400 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:07:55:014.435 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf6ac0 2025-02-13 Thu 02:07:55:014.442 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:07:55:014.451 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:07:55:014.460 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf7400 2025-02-13 Thu 02:07:55:014.465 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:07:55:014.474 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:07:55:015.928 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:07:55:015.990 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:07:55:016.008 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:07:55:016.034 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:07:55:016.064 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:07:55:016.143 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:07:55:016.162 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:07:55:016.173 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:07:55:016.187 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:07:55:016.199 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:07:55:016.215 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:07:55:016.228 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:07:55:016.254 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:07:55:016.277 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:07:55:016.300 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:07:59:932.802 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:07:59:932.956 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:07:59:933.015 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:07:59:933.060 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:07:59:933.108 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:07:59:934.318 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf9900 2025-02-13 Thu 02:07:59:934.340 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:07:59:934.369 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:07:59:934.395 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bfa240 2025-02-13 Thu 02:07:59:934.401 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:07:59:934.411 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:07:59:934.420 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bfab80 2025-02-13 Thu 02:07:59:934.426 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:07:59:934.435 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:07:59:934.444 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bfb4c0 2025-02-13 Thu 02:07:59:934.450 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:07:59:934.459 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:07:59:934.468 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf8fc0 2025-02-13 Thu 02:07:59:934.474 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:07:59:934.483 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:07:59:934.856 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:07:59:934.943 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:07:59:934.973 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:07:59:934.991 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:07:59:935.002 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:07:59:935.015 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:07:59:935.026 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:07:59:935.040 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:07:59:935.059 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:07:59:935.073 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:07:59:935.083 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:07:59:935.164 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:07:59:935.191 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:07:59:935.213 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:07:59:935.231 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:08:07:389.138 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 77 without seqno , total = 0 2025-02-13 Thu 02:09:07:390.723 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 78 without seqno , total = 0 2025-02-13 Thu 02:10:07:392.374 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 79 without seqno , total = 0 2025-02-13 Thu 02:10:47:836.728 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bf58b0 2025-02-13 Thu 02:10:47:836.789 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:836.845 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bf5140 2025-02-13 Thu 02:10:47:836.858 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:836.889 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bee070 2025-02-13 Thu 02:10:47:836.901 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:836.928 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bf4d40 2025-02-13 Thu 02:10:47:836.941 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:836.976 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-13 Thu 02:10:47:836.992 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bf4620 2025-02-13 Thu 02:10:47:837.002 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:870.718 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:10:47:870.820 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.46:60282::10.0.0.46:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-13 Thu 02:10:47:870.846 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:60282::10.0.0.46:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-13 Thu 02:10:47:870.855 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:60282::10.0.0.46:8086(-1):EvTcpClose 2025-02-13 Thu 02:10:47:871.350 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Processing scm::EvTcpClose in state Established 2025-02-13 Thu 02:10:47:871.365 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Established : EvTcpClose 2025-02-13 Thu 02:10:47:871.640 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Connect : Start Connect timer 10.0.0.35:8086 2025-02-13 Thu 02:10:47:871.735 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:10:47:872.654 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SANDESH: Send FAILED: 1739412647871473 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = cn-jenkins-deploy-platform-ansible-os-2728-1. collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3900432 calls = 411 average_bytes = 9490 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 169, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 32, _iter59->first = sent _iter59->second = 415, ] ] ] 2025-02-13 Thu 02:10:47:872.699 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:47:872.716 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:10:47:872.745 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SANDESH: Send FAILED: 1739412647871510 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.46:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:10:47:872.753 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:10:47:872.765 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2025-02-13 Thu 02:10:47:872.776 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:10:47:872.932 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SANDESH: Send FAILED: 1739412647871720 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3900432 calls = 411 average_bytes = 9490 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 169, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 32, _iter59->first = sent _iter59->second = 415, ] ] ] 2025-02-13 Thu 02:10:47:872.950 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:47:873.121 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-13 Thu 02:10:47:873.133 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bf6180 2025-02-13 Thu 02:10:47:873.145 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:873.187 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bf7400 2025-02-13 Thu 02:10:47:873.199 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:873.225 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bf6ac0 2025-02-13 Thu 02:10:47:873.237 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:873.258 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bf7d40 2025-02-13 Thu 02:10:47:873.270 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:873.293 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bf8680 2025-02-13 Thu 02:10:47:873.303 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:873.342 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:10:47:873.364 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:10:47:873.379 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:10:47:873.395 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:10:47:873.586 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300064380672, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:10:47:873.643 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300064380672, Pid 1]: SANDESH: Send FAILED: 1739412647873487 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.46:6379::Down.10.0.0.35:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:10:47:873.661 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300064380672, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:10:47:885.983 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.46:53200::10.0.0.35:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-13 Thu 02:10:47:891.495 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:53200::10.0.0.35:8086(-1) OnSessionEvent TCP Connected 2025-02-13 Thu 02:10:47:891.518 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:53200::10.0.0.35:8086(-1):EvTcpConnected 2025-02-13 Thu 02:10:47:894.814 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-13 Thu 02:10:47:894.855 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-13 Thu 02:10:47:894.907 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: ClientInit 2025-02-13 Thu 02:10:47:894.994 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2728-1.:contrail-query-engine:0:Database count 4 2025-02-13 Thu 02:10:47:895.177 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:47:895.196 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:10:47:895.225 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: SANDESH: Send FAILED: 1739412647894896 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.46:6379::Down.10.0.0.35:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:10:47:895.236 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:10:47:895.247 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:47:895.255 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-13 Thu 02:10:47:895.278 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:47:895.285 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-13 Thu 02:10:47:895.509 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: SANDESH: Send FAILED: 1739412647895163 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3900432 calls = 411 average_bytes = 9490 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 163 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 176, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 36, _iter59->first = sent _iter59->second = 415, ] ] ] 2025-02-13 Thu 02:10:47:895.528 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-13 Thu 02:10:47:903.447 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:10:47:903.522 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:10:47:903.542 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:10:47:903.559 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:10:47:903.584 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:10:47:903.617 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-13 Thu 02:10:47:903.626 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bf8fc0 2025-02-13 Thu 02:10:47:903.655 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:903.685 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bfb4c0 2025-02-13 Thu 02:10:47:903.697 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:903.719 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bf9900 2025-02-13 Thu 02:10:47:903.730 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:903.757 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bfab80 2025-02-13 Thu 02:10:47:903.769 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:903.792 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Disconnect status: -1 0x1bfa240 2025-02-13 Thu 02:10:47:903.802 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-13 Thu 02:10:47:904.226 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:47:904.247 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:10:47:904.281 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: SANDESH: Send FAILED: 1739412647903894 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:47:904.321 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:10:47:904.716 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:10:47:904.754 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:10:47:904.769 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:10:47:904.808 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:10:47:904.835 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:10:47:939.954 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.46:53200::10.0.0.35:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2025-02-13 Thu 02:10:47:939.999 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.46:53200::10.0.0.35:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-13 Thu 02:10:47:940.025 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:53200::10.0.0.35:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-13 Thu 02:10:47:940.034 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:53200::10.0.0.35:8086(-1):EvTcpClose 2025-02-13 Thu 02:10:47:941.914 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-13 Thu 02:10:47:941.937 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: ClientInit : EvTcpClose 2025-02-13 Thu 02:10:47:942.104 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Return to idle with 1000 2025-02-13 Thu 02:10:47:942.116 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Idle 2025-02-13 Thu 02:10:47:942.187 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:10:47:942.394 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: SANDESH: Send FAILED: 1739412647942041 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3902005 calls = 412 average_bytes = 9470 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 184, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 39, _iter59->first = sent _iter59->second = 416, ] ] ] 2025-02-13 Thu 02:10:47:942.433 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:47:942.451 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:10:47:942.482 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: SANDESH: Send FAILED: 1739412647942078 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:47:942.492 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:10:47:942.507 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:10:47:942.523 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:10:47:942.689 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: SANDESH: Send FAILED: 1739412647942174 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3902005 calls = 412 average_bytes = 9470 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 184, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 39, _iter59->first = sent _iter59->second = 416, ] ] ] 2025-02-13 Thu 02:10:47:942.711 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:48:944.139 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-13 Thu 02:10:48:944.365 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-02-13 Thu 02:10:48:944.525 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:10:48:944.561 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SANDESH: Send FAILED: 1739412648944218 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:48:944.571 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:10:48:944.582 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:10:48:944.747 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SANDESH: Send FAILED: 1739412648944510 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3902005 calls = 412 average_bytes = 9470 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 184, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 42, _iter59->first = sent _iter59->second = 416, ] ] ] 2025-02-13 Thu 02:10:48:944.774 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:48:946.999 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.46:46522::10.0.0.12:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-13 Thu 02:10:48:947.030 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:46522::10.0.0.12:8086(-1) OnSessionEvent TCP Connected 2025-02-13 Thu 02:10:48:947.039 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:46522::10.0.0.12:8086(-1):EvTcpConnected 2025-02-13 Thu 02:10:48:947.097 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-13 Thu 02:10:48:947.119 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-13 Thu 02:10:48:947.161 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: ClientInit 2025-02-13 Thu 02:10:48:947.185 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2728-1.:contrail-query-engine:0:Database count 5 2025-02-13 Thu 02:10:48:947.280 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:48:947.290 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:10:48:947.328 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: SANDESH: Send FAILED: 1739412648947153 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:48:947.340 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:10:48:947.354 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:48:947.364 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-13 Thu 02:10:48:947.386 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:48:947.396 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-13 Thu 02:10:48:947.584 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: SANDESH: Send FAILED: 1739412648947268 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3902005 calls = 412 average_bytes = 9470 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 185, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 44, _iter59->first = sent _iter59->second = 416, ] ] ] 2025-02-13 Thu 02:10:48:947.602 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-13 Thu 02:10:48:948.370 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:46522::10.0.0.12:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-13 Thu 02:10:48:948.391 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:46522::10.0.0.12:8086(-1):EvTcpClose 2025-02-13 Thu 02:10:48:948.440 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-13 Thu 02:10:48:948.462 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: ClientInit : EvTcpClose 2025-02-13 Thu 02:10:48:948.585 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Return to idle with 1000 2025-02-13 Thu 02:10:48:948.595 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Idle 2025-02-13 Thu 02:10:48:948.663 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:10:48:948.832 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SANDESH: Send FAILED: 1739412648948542 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3903578 calls = 413 average_bytes = 9451 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 185, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 46, _iter59->first = sent _iter59->second = 417, ] ] ] 2025-02-13 Thu 02:10:48:948.849 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:48:948.861 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:10:48:948.884 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SANDESH: Send FAILED: 1739412648948569 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:48:948.893 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:10:48:948.902 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:10:48:948.930 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:10:48:949.081 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SANDESH: Send FAILED: 1739412648948651 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3903578 calls = 413 average_bytes = 9451 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 185, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 46, _iter59->first = sent _iter59->second = 417, ] ] ] 2025-02-13 Thu 02:10:48:949.097 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:49:950.301 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-13 Thu 02:10:49:950.564 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Connect : Start Connect timer 10.0.0.46:8086 2025-02-13 Thu 02:10:49:950.682 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:10:49:950.713 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: SANDESH: Send FAILED: 1739412649950398 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:49:950.736 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:10:49:950.747 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:10:49:950.919 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: SANDESH: Send FAILED: 1739412649950669 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3903578 calls = 413 average_bytes = 9451 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 185, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 49, _iter59->first = sent _iter59->second = 417, ] ] ] 2025-02-13 Thu 02:10:49:950.937 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:49:958.181 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.46:37456::10.0.0.46:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-13 Thu 02:10:49:958.216 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:37456::10.0.0.46:8086(-1) OnSessionEvent TCP Connected 2025-02-13 Thu 02:10:49:958.225 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:37456::10.0.0.46:8086(-1):EvTcpConnected 2025-02-13 Thu 02:10:49:960.144 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-13 Thu 02:10:49:960.169 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-13 Thu 02:10:49:960.214 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: ClientInit 2025-02-13 Thu 02:10:49:960.235 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2728-1.:contrail-query-engine:0:Database count 6 2025-02-13 Thu 02:10:49:960.362 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:49:960.375 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:10:49:960.402 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SANDESH: Send FAILED: 1739412649960205 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:49:960.411 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:10:49:960.421 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:49:960.428 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-13 Thu 02:10:49:960.441 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:49:960.448 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-13 Thu 02:10:49:960.602 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SANDESH: Send FAILED: 1739412649960350 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3903578 calls = 413 average_bytes = 9451 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 186, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 51, _iter59->first = sent _iter59->second = 417, ] ] ] 2025-02-13 Thu 02:10:49:960.623 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-13 Thu 02:10:49:962.141 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:37456::10.0.0.46:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-13 Thu 02:10:49:962.161 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:37456::10.0.0.46:8086(-1):EvTcpClose 2025-02-13 Thu 02:10:49:964.543 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-13 Thu 02:10:49:964.561 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: ClientInit : EvTcpClose 2025-02-13 Thu 02:10:49:964.728 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Return to idle with 1000 2025-02-13 Thu 02:10:49:964.739 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Idle 2025-02-13 Thu 02:10:49:964.808 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:10:49:965.014 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: SANDESH: Send FAILED: 1739412649964671 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3905151 calls = 414 average_bytes = 9432 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 186, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 53, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-02-13 Thu 02:10:49:965.040 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:49:965.052 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:10:49:965.077 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: SANDESH: Send FAILED: 1739412649964702 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:49:965.085 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:10:49:965.095 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:10:49:965.105 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:10:49:965.255 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: SANDESH: Send FAILED: 1739412649964796 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3905151 calls = 414 average_bytes = 9432 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 186, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 53, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-02-13 Thu 02:10:49:965.272 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:50:965.725 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-13 Thu 02:10:50:965.965 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Connect : Start Connect timer 10.0.0.35:8086 2025-02-13 Thu 02:10:50:966.077 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:10:50:966.109 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: SANDESH: Send FAILED: 1739412650965821 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:50:966.120 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:10:50:966.134 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:10:50:966.297 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: SANDESH: Send FAILED: 1739412650966062 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3905151 calls = 414 average_bytes = 9432 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 186, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 56, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-02-13 Thu 02:10:50:966.336 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:50:968.848 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.46:50890::10.0.0.35:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-13 Thu 02:10:50:968.882 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:50890::10.0.0.35:8086(-1) OnSessionEvent TCP Connected 2025-02-13 Thu 02:10:50:968.891 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:50890::10.0.0.35:8086(-1):EvTcpConnected 2025-02-13 Thu 02:10:50:969.257 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-13 Thu 02:10:50:969.293 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-13 Thu 02:10:50:969.348 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: ClientInit 2025-02-13 Thu 02:10:50:969.374 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2728-1.:contrail-query-engine:0:Database count 7 2025-02-13 Thu 02:10:50:969.491 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:50:969.503 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:10:50:969.530 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: SANDESH: Send FAILED: 1739412650969338 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:50:969.555 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:10:50:969.566 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:50:969.574 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-13 Thu 02:10:50:969.586 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:50:969.593 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-13 Thu 02:10:50:969.757 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: SANDESH: Send FAILED: 1739412650969478 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 7 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3905151 calls = 414 average_bytes = 9432 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 187, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 58, _iter59->first = sent _iter59->second = 418, ] ] ] 2025-02-13 Thu 02:10:50:969.774 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-13 Thu 02:10:52:870.829 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:10:52:871.045 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:10:52:873.377 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:10:52:873.454 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:10:52:873.472 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:10:52:873.507 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:10:52:873.545 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:10:52:873.583 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:10:52:873.600 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:10:52:873.615 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:10:52:903.552 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:10:52:903.725 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:10:52:903.772 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:10:52:903.811 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:10:52:903.866 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:10:52:904.262 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:10:52:904.349 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:10:52:904.371 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:10:52:904.388 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:10:52:904.416 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:10:52:904.765 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:10:52:904.817 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:10:52:904.856 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:10:52:904.892 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:10:52:904.943 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:10:52:905.183 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:10:52:905.203 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:10:52:905.219 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:10:52:905.235 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:10:52:905.264 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:10:57:871.117 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:10:57:871.338 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:10:57:873.488 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:10:57:873.568 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:10:57:873.613 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:10:57:873.659 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:10:57:873.674 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:10:57:873.705 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:10:57:873.745 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:10:57:873.761 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:10:57:904.375 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:10:57:904.694 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:10:57:904.775 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:10:57:904.830 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:10:57:904.880 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:10:57:905.015 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:10:57:905.064 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:10:57:905.157 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:10:57:905.189 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:10:57:905.239 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:10:57:905.257 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:10:57:905.324 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:10:57:905.379 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:10:57:905.423 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:10:57:905.485 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:10:57:905.635 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:10:57:905.672 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:10:57:905.697 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:10:57:905.720 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:10:57:905.739 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:10:58:969.499 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: 10.0.0.35:8086 EvConnectTimerExpired in state ClientInit 2025-02-13 Thu 02:10:58:969.606 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300676757248, Pid 1]: Processing scm::EvConnectTimerExpired in state ClientInit 2025-02-13 Thu 02:10:58:969.645 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300676757248, Pid 1]: ClientInit : EvConnectTimerExpired 2025-02-13 Thu 02:10:58:969.895 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300676757248, Pid 1]: Return to idle with 1000 2025-02-13 Thu 02:10:58:969.908 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300676757248, Pid 1]: Idle 2025-02-13 Thu 02:10:58:969.981 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300676757248, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:10:58:970.162 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300676757248, Pid 1]: SANDESH: Send FAILED: 1739412658969758 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 7 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3906724 calls = 415 average_bytes = 9413 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 187, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 60, _iter59->first = sent _iter59->second = 419, ] ] ] 2025-02-13 Thu 02:10:58:970.194 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300676757248, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:58:970.207 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300676757248, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:10:58:970.233 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300676757248, Pid 1]: SANDESH: Send FAILED: 1739412658969797 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Down description = ClientInit : EvConnectTimerExpired ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:58:970.241 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300676757248, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:10:58:970.251 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300676757248, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:10:58:970.285 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300676757248, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:10:58:970.451 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300676757248, Pid 1]: SANDESH: Send FAILED: 1739412658969969 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 7 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3906724 calls = 415 average_bytes = 9413 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 187, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 60, _iter59->first = sent _iter59->second = 419, ] ] ] 2025-02-13 Thu 02:10:58:970.470 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300676757248, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:59:970.085 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-13 Thu 02:10:59:970.500 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-02-13 Thu 02:10:59:970.700 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:10:59:970.799 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: SANDESH: Send FAILED: 1739412659970192 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:59:970.813 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:10:59:970.826 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:10:59:971.006 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: SANDESH: Send FAILED: 1739412659970671 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 7 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3906724 calls = 415 average_bytes = 9413 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 187, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 63, _iter59->first = sent _iter59->second = 419, ] ] ] 2025-02-13 Thu 02:10:59:971.041 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:10:59:973.282 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.46:46848::10.0.0.12:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-13 Thu 02:10:59:973.318 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:46848::10.0.0.12:8086(-1) OnSessionEvent TCP Connected 2025-02-13 Thu 02:10:59:973.329 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:46848::10.0.0.12:8086(-1):EvTcpConnected 2025-02-13 Thu 02:10:59:973.396 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300651575040, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-13 Thu 02:10:59:973.431 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300651575040, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-13 Thu 02:10:59:973.490 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300651575040, Pid 1]: ClientInit 2025-02-13 Thu 02:10:59:973.515 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300651575040, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2728-1.:contrail-query-engine:0:Database count 8 2025-02-13 Thu 02:10:59:973.637 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300651575040, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:59:973.654 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300651575040, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:10:59:973.690 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300651575040, Pid 1]: SANDESH: Send FAILED: 1739412659973479 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:10:59:973.702 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300651575040, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:10:59:973.713 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300651575040, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:59:973.721 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300651575040, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-13 Thu 02:10:59:973.744 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300651575040, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:10:59:973.752 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300651575040, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-13 Thu 02:10:59:973.903 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300651575040, Pid 1]: SANDESH: Send FAILED: 1739412659973622 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 8 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3906724 calls = 415 average_bytes = 9413 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 188, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 65, _iter59->first = sent _iter59->second = 419, ] ] ] 2025-02-13 Thu 02:10:59:973.919 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300651575040, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-13 Thu 02:11:00:949.033 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.46:46848::10.0.0.12:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2025-02-13 Thu 02:11:00:949.087 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.46:46848::10.0.0.12:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-13 Thu 02:11:00:949.113 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:46848::10.0.0.12:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-13 Thu 02:11:00:949.123 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:46848::10.0.0.12:8086(-1):EvTcpClose 2025-02-13 Thu 02:11:00:949.669 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-13 Thu 02:11:00:949.683 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: ClientInit : EvTcpClose 2025-02-13 Thu 02:11:00:949.855 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Return to idle with 1000 2025-02-13 Thu 02:11:00:949.870 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Idle 2025-02-13 Thu 02:11:00:949.956 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:00:950.132 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: SANDESH: Send FAILED: 1739412660949787 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 8 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 190, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 67, _iter59->first = sent _iter59->second = 420, ] ] ] 2025-02-13 Thu 02:11:00:950.152 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:00:950.164 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:00:950.190 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: SANDESH: Send FAILED: 1739412660949822 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:11:00:950.201 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:11:00:950.211 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:11:00:950.222 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:00:950.427 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: SANDESH: Send FAILED: 1739412660949944 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 8 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 190, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 67, _iter59->first = sent _iter59->second = 420, ] ] ] 2025-02-13 Thu 02:11:00:950.449 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300760581888, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:01:949.996 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-13 Thu 02:11:01:950.282 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Connect : Start Connect timer 10.0.0.46:8086 2025-02-13 Thu 02:11:01:950.420 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:11:01:950.466 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: SANDESH: Send FAILED: 1739412661950097 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:11:01:950.477 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:11:01:950.488 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:11:01:950.662 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: SANDESH: Send FAILED: 1739412661950403 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 8 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 190, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 70, _iter59->first = sent _iter59->second = 420, ] ] ] 2025-02-13 Thu 02:11:01:950.681 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:01:950.719 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-13 Thu 02:11:01:950.734 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: -(-1):EvTcpConnectFail 2025-02-13 Thu 02:11:01:951.285 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-13 Thu 02:11:01:951.296 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Connect : EvTcpConnectFail 2025-02-13 Thu 02:11:01:951.428 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-13 Thu 02:11:01:951.465 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Idle 2025-02-13 Thu 02:11:01:951.530 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:01:951.709 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: SANDESH: Send FAILED: 1739412661951379 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 8 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 190, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 72, _iter59->first = sent _iter59->second = 420, ] ] ] 2025-02-13 Thu 02:11:01:951.727 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:01:951.739 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:01:951.763 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: SANDESH: Send FAILED: 1739412661951404 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:11:01:951.775 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:11:01:951.785 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:11:01:951.795 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:01:951.970 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: SANDESH: Send FAILED: 1739412661951520 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 8 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 191, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 72, _iter59->first = sent _iter59->second = 420, ] ] ] 2025-02-13 Thu 02:11:01:951.988 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:02:871.433 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:11:02:871.666 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:11:02:873.613 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:11:02:873.721 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:11:02:873.743 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:11:02:873.803 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:11:02:873.823 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:11:02:873.861 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:11:02:873.900 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:11:02:873.917 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:11:02:905.096 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:11:02:905.277 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:11:02:905.334 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:11:02:905.379 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:11:02:905.424 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:11:02:905.729 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:11:02:905.781 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:11:02:905.848 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:11:02:905.903 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:11:02:905.927 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:11:02:905.942 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:11:02:905.957 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:11:02:905.969 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:11:02:906.002 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:11:02:906.036 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:11:02:906.233 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:11:02:906.270 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:11:02:906.286 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:11:02:906.338 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:11:02:906.355 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:11:07:392.969 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 80 without seqno , total = 0 2025-02-13 Thu 02:11:07:393.077 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:07:393.210 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: SANDESH: Send FAILED: 1739412667392830 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 495 dequeues = 495 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 535 dequeues = 535 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739412661951464 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739412661951793 ] max_sm_queue_count = 6 ] 2025-02-13 Thu 02:11:07:393.228 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:07:393.242 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:07:393.457 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: SANDESH: Send FAILED: 1739412667392948 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 8 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 191, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 75, _iter59->first = sent _iter59->second = 420, ] ] ] 2025-02-13 Thu 02:11:07:393.490 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:07:871.743 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:11:07:871.997 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:11:07:873.768 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:11:07:873.868 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:11:07:873.889 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:11:07:873.940 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:11:07:873.954 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:11:07:873.986 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:11:07:874.024 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:11:07:874.041 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:11:07:905.815 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:11:07:906.007 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:11:07:906.047 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:11:07:906.077 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:11:07:906.104 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:11:07:906.284 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:11:07:906.370 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:11:07:906.405 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:11:07:906.438 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:11:07:906.502 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:11:07:906.535 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:11:07:906.575 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:11:07:906.596 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:11:07:906.611 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:11:07:906.625 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:11:07:906.640 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:11:07:906.757 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:11:07:906.774 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:11:07:906.788 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:11:07:906.877 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:11:12:872.081 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:11:12:872.290 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:11:12:873.903 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:11:12:873.973 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:11:12:873.991 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:11:12:874.030 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:11:12:874.055 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:11:12:874.091 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:11:12:874.143 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:11:12:874.161 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:11:12:906.571 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:11:12:906.741 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:11:12:906.810 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:11:12:906.846 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:11:12:906.882 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:11:12:906.915 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:11:12:906.955 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:11:12:906.989 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:11:12:907.026 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:11:12:907.062 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:11:12:907.309 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:11:12:907.355 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:11:12:907.375 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:11:12:907.405 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:11:12:907.421 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:11:12:907.434 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:11:12:907.449 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:11:12:907.465 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:11:12:907.480 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:11:12:911.212 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:11:17:872.858 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:11:17:873.087 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:11:17:874.060 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:11:17:874.132 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:11:17:874.193 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:11:17:874.215 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:11:17:874.227 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:11:17:874.260 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:11:17:874.342 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:11:17:874.360 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:11:17:907.405 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:11:17:907.580 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:11:17:907.642 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:11:17:907.686 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:11:17:907.730 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:11:17:907.769 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:11:17:907.804 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:11:17:907.942 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:11:17:907.997 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:11:17:908.147 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:11:17:908.191 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:11:17:908.212 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:11:17:908.237 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:11:17:908.259 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:11:17:908.276 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:11:17:908.293 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:11:17:908.309 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:11:17:908.338 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:11:17:912.434 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:11:17:912.880 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:11:22:873.179 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:11:22:873.441 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:11:22:874.229 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:11:22:874.298 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:11:22:874.378 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:11:22:874.399 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:11:22:874.411 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:11:22:874.453 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:11:22:874.466 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:11:22:874.501 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:11:22:908.256 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:11:22:908.479 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:11:22:908.527 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:11:22:908.570 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:11:22:908.604 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:11:22:908.636 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:11:22:908.670 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:11:22:908.706 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:11:22:908.739 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:11:22:909.029 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:11:22:909.104 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:11:22:909.129 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:11:22:909.148 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:11:22:909.169 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:11:22:909.194 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:11:22:909.216 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:11:22:909.236 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:11:22:909.258 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:11:22:912.974 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:11:22:913.560 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:11:27:873.522 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:11:27:873.720 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:11:27:874.421 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:11:27:874.487 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:11:27:874.534 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:11:27:874.554 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:11:27:874.567 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:11:27:874.599 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:11:27:874.633 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:11:27:874.649 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:11:27:909.128 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:11:27:909.289 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:11:27:909.336 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:11:27:909.371 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:11:27:909.401 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:11:27:909.433 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:11:27:909.461 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:11:27:909.493 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:11:27:909.522 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:11:27:909.748 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:11:27:909.778 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:11:27:909.795 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:11:27:909.809 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:11:27:909.824 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:11:27:909.838 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:11:27:909.866 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:11:27:909.882 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:11:27:909.895 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:11:27:913.602 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:11:27:914.037 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:11:31:951.585 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-13 Thu 02:11:31:951.841 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Connect : Start Connect timer 10.0.0.35:8086 2025-02-13 Thu 02:11:31:951.958 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:11:31:951.989 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: SANDESH: Send FAILED: 1739412691951661 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:11:31:952.000 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:11:31:952.011 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:11:31:952.181 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: SANDESH: Send FAILED: 1739412691951945 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 8 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 191, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 77, _iter59->first = sent _iter59->second = 420, ] ] ] 2025-02-13 Thu 02:11:31:952.209 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:31:958.546 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-13 Thu 02:11:31:958.581 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: -(-1):EvTcpConnectFail 2025-02-13 Thu 02:11:31:960.349 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-13 Thu 02:11:31:960.372 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Connect : EvTcpConnectFail 2025-02-13 Thu 02:11:31:960.585 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-13 Thu 02:11:31:960.629 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Idle 2025-02-13 Thu 02:11:31:960.704 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:31:960.879 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: SANDESH: Send FAILED: 1739412691960502 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 8 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 191, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 79, _iter59->first = sent _iter59->second = 420, ] ] ] 2025-02-13 Thu 02:11:31:960.911 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:31:960.923 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:31:960.949 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: SANDESH: Send FAILED: 1739412691960549 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:11:31:960.957 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:11:31:960.968 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:11:31:960.978 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:31:961.126 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: SANDESH: Send FAILED: 1739412691960692 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 8 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 22 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 192, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 79, _iter59->first = sent _iter59->second = 420, ] ] ] 2025-02-13 Thu 02:11:31:961.144 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300060182272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:32:873.837 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:11:32:874.012 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:11:32:874.571 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:11:32:874.637 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:11:32:874.662 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:11:32:874.705 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:11:32:874.719 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:11:32:874.750 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:11:32:874.788 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:11:32:874.804 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:11:32:909.821 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:11:32:909.972 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:11:32:910.015 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:11:32:910.055 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:11:32:910.086 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:11:32:910.113 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:11:32:910.139 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:11:32:910.182 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:11:32:910.211 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:11:32:910.402 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:11:32:910.438 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:11:32:910.455 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:11:32:910.471 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:11:32:910.487 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:11:32:910.502 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:11:32:910.533 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:11:32:910.548 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:11:32:910.563 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:11:32:914.100 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:11:32:914.590 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:11:37:397.231 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:37:397.407 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SANDESH: Send FAILED: 1739412697397016 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 502 dequeues = 502 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 545 dequeues = 545 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739412691960627 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739412691960977 ] max_sm_queue_count = 6 ] 2025-02-13 Thu 02:11:37:397.433 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:37:397.450 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:37:397.728 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SANDESH: Send FAILED: 1739412697397186 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 8 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 22 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 192, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 82, _iter59->first = sent _iter59->second = 420, ] ] ] 2025-02-13 Thu 02:11:37:397.766 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:11:37:874.089 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:11:37:874.297 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:11:37:874.704 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:11:37:874.771 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:11:37:874.788 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:11:37:874.837 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:11:37:874.852 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:11:37:874.883 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:11:37:874.922 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:11:37:874.952 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:11:37:910.474 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:11:37:910.713 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:11:37:910.762 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:11:37:910.794 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:11:37:910.832 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:11:37:910.868 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:11:37:910.905 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:11:37:910.940 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:11:37:910.974 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:11:37:911.027 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:11:37:911.139 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:11:37:911.159 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:11:37:911.174 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:11:37:911.191 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:11:37:911.207 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:11:37:911.222 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:11:37:911.239 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:11:37:911.438 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:11:37:914.641 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:11:37:915.021 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:11:42:388.352 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300634789632, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:11:42:388.441 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300634789632, Pid 1]: SANDESH: Send FAILED: 1739412702388285 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-02-13 Thu 02:11:42:388.467 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300634789632, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-02-13 Thu 02:11:42:874.383 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:11:42:874.545 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:11:42:874.813 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:11:42:874.896 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:11:42:874.918 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:11:42:874.980 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:11:42:874.995 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:11:42:875.028 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:11:42:875.064 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:11:42:875.081 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:11:42:911.102 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:11:42:911.256 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:11:42:911.295 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:11:42:911.336 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:11:42:911.367 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:11:42:911.403 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:11:42:911.437 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:11:42:911.474 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:11:42:911.500 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:11:42:911.704 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:11:42:911.738 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:11:42:911.815 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:11:42:911.921 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:11:42:911.970 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:11:42:911.994 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:11:42:912.013 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:11:42:912.033 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:11:42:912.052 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:11:42:915.087 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:11:42:915.663 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:11:47:874.622 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:11:47:874.786 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:11:47:874.930 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:11:47:874.980 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:11:47:875.013 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:11:47:875.057 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:11:47:875.092 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:11:47:875.128 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:11:47:875.166 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:11:47:875.182 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:11:47:911.782 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:11:47:912.010 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:11:47:912.052 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:11:47:912.083 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:11:47:912.112 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:11:47:912.148 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:11:47:912.180 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:11:47:912.209 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:11:47:912.239 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:11:47:912.388 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:11:47:912.417 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:11:47:912.451 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:11:47:912.476 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:11:47:912.503 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:11:47:912.659 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:11:47:912.696 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:11:47:912.728 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:11:47:914.048 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:11:47:915.714 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:11:47:916.133 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:11:52:874.884 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:11:52:875.132 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:11:52:875.191 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:11:52:875.252 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:11:52:875.328 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:11:52:875.357 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:11:52:875.377 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:11:52:875.430 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:11:52:875.484 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:11:52:875.510 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:11:52:912.491 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:11:52:912.697 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:11:52:912.777 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:11:52:912.827 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:11:52:912.873 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:11:52:912.934 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:11:52:912.980 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:11:52:913.071 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:11:52:913.143 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:11:52:913.219 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:11:52:913.247 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:11:52:913.270 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:11:52:913.292 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:11:52:913.367 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:11:52:913.439 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:11:52:913.462 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:11:52:914.108 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:11:52:914.582 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:11:52:916.194 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:11:52:916.627 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:11:57:875.215 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:11:57:875.421 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:11:57:875.458 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:11:57:875.522 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:11:57:875.582 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:11:57:875.606 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:11:57:875.623 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:11:57:875.667 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:11:57:875.718 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:11:57:875.742 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:11:57:913.218 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:11:57:913.424 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:11:57:913.486 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:11:57:913.523 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:11:57:913.557 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:11:57:913.588 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:11:57:913.627 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:11:57:913.658 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:11:57:913.737 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:11:57:913.783 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:11:57:913.806 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:11:57:913.821 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:11:57:913.835 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:11:57:913.947 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:11:57:914.011 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:11:57:914.036 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:11:57:914.634 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:11:57:914.934 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:11:57:916.681 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:11:57:916.983 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:12:01:963.519 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-13 Thu 02:12:01:963.775 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-02-13 Thu 02:12:01:965.270 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:12:01:965.337 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SANDESH: Send FAILED: 1739412721963609 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:12:01:965.401 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:12:01:965.414 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:12:01:965.572 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SANDESH: Send FAILED: 1739412721965244 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 8 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 47 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 22 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 192, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 85, _iter59->first = sent _iter59->second = 420, ] ] ] 2025-02-13 Thu 02:12:01:965.590 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:12:01:965.637 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-13 Thu 02:12:01:965.647 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: -(-1):EvTcpConnectFail 2025-02-13 Thu 02:12:01:968.385 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-13 Thu 02:12:01:968.402 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Connect : EvTcpConnectFail 2025-02-13 Thu 02:12:01:968.546 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-13 Thu 02:12:01:968.588 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Idle 2025-02-13 Thu 02:12:01:968.656 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:01:968.962 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: SANDESH: Send FAILED: 1739412721968488 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 8 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 22 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 192, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 87, _iter59->first = sent _iter59->second = 420, ] ] ] 2025-02-13 Thu 02:12:01:968.988 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:12:01:969.001 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:01:969.028 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: SANDESH: Send FAILED: 1739412721968521 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:12:01:969.039 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:12:01:969.049 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:12:01:969.059 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:01:969.209 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: SANDESH: Send FAILED: 1739412721968644 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 8 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 23 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 193, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 87, _iter59->first = sent _iter59->second = 420, ] ] ] 2025-02-13 Thu 02:12:01:969.225 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:12:02:875.512 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:12:02:875.712 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:12:02:875.752 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:12:02:875.796 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:12:02:875.866 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:12:02:875.886 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:12:02:875.898 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:12:02:875.932 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:12:02:875.970 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:12:02:875.988 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:12:02:913.811 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:12:02:913.968 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:12:02:914.006 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:12:02:914.046 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:12:02:914.077 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:12:02:914.113 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:12:02:914.151 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:12:02:914.182 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:12:02:914.383 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:12:02:914.422 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:12:02:914.438 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:12:02:914.451 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:12:02:914.466 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:12:02:914.507 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:12:02:914.588 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:12:02:914.605 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:12:02:914.964 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:12:02:915.344 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:12:02:917.024 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:12:02:917.383 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:12:07:400.707 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 81 without seqno , total = 0 2025-02-13 Thu 02:12:07:400.771 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:07:400.848 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SANDESH: Send FAILED: 1739412727400574 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 510 dequeues = 510 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 556 dequeues = 556 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739412721968586 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739412721969058 ] max_sm_queue_count = 6 ] 2025-02-13 Thu 02:12:07:400.862 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:12:07:400.873 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:07:401.031 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SANDESH: Send FAILED: 1739412727400687 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 8 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 50 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 23 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 193, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 90, _iter59->first = sent _iter59->second = 420, ] ] ] 2025-02-13 Thu 02:12:07:401.060 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:12:07:879.385 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:12:07:879.551 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:12:07:879.593 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:12:07:879.626 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:12:07:879.660 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:12:07:879.708 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:12:07:879.739 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:12:07:879.754 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:12:07:879.767 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:12:07:879.781 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:12:07:917.048 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:12:07:917.192 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:12:07:917.231 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:12:07:917.264 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:12:07:917.297 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:12:07:917.338 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:12:07:917.372 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:12:07:917.404 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:12:07:917.440 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:12:07:917.498 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:12:07:917.527 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:12:07:917.539 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:12:07:919.886 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:12:07:919.938 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:12:07:919.956 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:12:07:919.973 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:12:07:919.988 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:12:07:920.004 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:12:07:920.019 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:12:07:920.034 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:12:12:879.780 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:12:12:879.991 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:12:12:880.022 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:12:12:880.062 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:12:12:880.098 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:12:12:880.131 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:12:12:880.170 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:12:12:880.188 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:12:12:880.202 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:12:12:880.216 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:12:12:917.558 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:12:12:917.692 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:12:12:918.036 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:12:12:918.072 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:12:12:919.935 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:12:12:920.032 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:12:12:920.067 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:12:12:920.104 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:12:12:920.141 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:12:12:920.171 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:12:12:920.206 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:12:12:920.236 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:12:12:920.354 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:12:12:920.378 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:12:12:920.392 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:12:12:920.552 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:12:12:920.571 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:12:12:920.596 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:12:12:920.611 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:12:12:920.690 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:12:17:880.066 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:12:17:880.267 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:12:17:880.297 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:12:17:880.353 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:12:17:880.389 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:12:17:880.421 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:12:17:880.461 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:12:17:880.479 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:12:17:880.493 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:12:17:880.507 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:12:17:918.112 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:12:17:918.261 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:12:17:918.499 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:12:17:918.530 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:12:17:920.401 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:12:17:920.483 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:12:17:920.518 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:12:17:920.584 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:12:17:920.628 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:12:17:920.661 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:12:17:920.693 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:12:17:920.745 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:12:17:920.768 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:12:17:920.780 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:12:17:920.924 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:12:17:920.955 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:12:17:921.028 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:12:17:921.063 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:12:17:921.080 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:12:17:921.184 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:12:22:880.352 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:12:22:880.554 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:12:22:880.587 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:12:22:880.629 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:12:22:880.677 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:12:22:880.710 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:12:22:880.761 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:12:22:880.780 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:12:22:880.795 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:12:22:880.808 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:12:22:918.564 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:12:22:918.746 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:12:22:919.173 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:12:22:919.233 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:12:22:920.785 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:12:22:920.866 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:12:22:920.952 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:12:22:921.004 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:12:22:921.048 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:12:22:921.095 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:12:22:921.132 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:12:22:921.179 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:12:22:921.219 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:12:22:921.235 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:12:22:921.246 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:12:22:921.349 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:12:22:921.369 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:12:22:921.409 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:12:22:921.465 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:12:22:921.582 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:12:27:880.629 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:12:27:880.815 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf58b0 2025-02-13 Thu 02:12:27:880.854 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:12:27:880.896 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:12:27:880.940 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf5140 2025-02-13 Thu 02:12:27:880.957 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bee070 2025-02-13 Thu 02:12:27:880.969 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:12:27:881.000 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:12:27:881.035 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4d40 2025-02-13 Thu 02:12:27:881.050 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf4620 2025-02-13 Thu 02:12:27:919.265 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:12:27:919.418 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:12:27:919.725 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfb4c0 2025-02-13 Thu 02:12:27:919.766 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf9900 2025-02-13 Thu 02:12:27:921.220 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:12:27:921.308 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:12:27:921.353 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:12:27:921.399 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:12:27:921.440 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:12:27:921.478 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:12:27:921.509 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:12:27:921.559 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8fc0 2025-02-13 Thu 02:12:27:921.669 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfa240 2025-02-13 Thu 02:12:27:921.699 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bfab80 2025-02-13 Thu 02:12:27:921.712 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:12:27:921.758 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6180 2025-02-13 Thu 02:12:27:921.776 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf6ac0 2025-02-13 Thu 02:12:27:922.777 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7d40 2025-02-13 Thu 02:12:27:922.825 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf8680 2025-02-13 Thu 02:12:27:922.842 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: -1 0x1bf7400 2025-02-13 Thu 02:12:31:970.125 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-13 Thu 02:12:31:970.431 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Connect : Start Connect timer 10.0.0.46:8086 2025-02-13 Thu 02:12:31:970.561 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:12:31:970.596 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SANDESH: Send FAILED: 1739412751970230 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.46:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:12:31:970.606 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:12:31:970.617 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:12:31:970.792 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SANDESH: Send FAILED: 1739412751970547 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 8 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.46:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 52 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 23 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 193, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 92, _iter59->first = sent _iter59->second = 420, ] ] ] 2025-02-13 Thu 02:12:31:970.819 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:12:31:972.183 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-13 Thu 02:12:31:972.202 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: -(-1):EvTcpConnectFail 2025-02-13 Thu 02:12:31:973.699 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-13 Thu 02:12:31:973.726 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Connect : EvTcpConnectFail 2025-02-13 Thu 02:12:31:973.888 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-13 Thu 02:12:31:973.927 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Idle 2025-02-13 Thu 02:12:31:973.997 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:31:974.175 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: SANDESH: Send FAILED: 1739412751973821 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 8 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 53 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 23 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 193, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 94, _iter59->first = sent _iter59->second = 420, ] ] ] 2025-02-13 Thu 02:12:31:974.225 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:12:31:974.237 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:31:974.263 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: SANDESH: Send FAILED: 1739412751973856 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:12:31:974.272 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:12:31:974.283 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:12:31:974.294 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:31:974.644 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: SANDESH: Send FAILED: 1739412751973986 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 8 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 53 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 24 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 194, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 94, _iter59->first = sent _iter59->second = 420, ] ] ] 2025-02-13 Thu 02:12:31:974.675 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:12:32:887.888 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf58b0 2025-02-13 Thu 02:12:32:888.091 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf5140 2025-02-13 Thu 02:12:32:888.145 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bee070 2025-02-13 Thu 02:12:32:888.187 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4d40 2025-02-13 Thu 02:12:32:888.237 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf4620 2025-02-13 Thu 02:12:32:888.292 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf58b0 2025-02-13 Thu 02:12:32:888.301 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:888.342 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:888.360 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf5140 2025-02-13 Thu 02:12:32:888.369 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:888.383 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:888.395 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bee070 2025-02-13 Thu 02:12:32:888.403 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:888.416 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:888.429 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf4d40 2025-02-13 Thu 02:12:32:888.438 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:888.450 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:888.462 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf4620 2025-02-13 Thu 02:12:32:888.471 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:888.482 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:895.917 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:895.965 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:895.983 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:895.995 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:896.010 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:896.021 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:896.036 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:896.046 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:896.061 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:896.071 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:896.100 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:897.299 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:32:897.357 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SANDESH: Send FAILED: 1739412752896134 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.35:6379::Down.10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:12:32:897.371 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:12:32:903.934 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:904.008 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:904.029 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:904.048 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:919.806 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfb4c0 2025-02-13 Thu 02:12:32:919.975 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf9900 2025-02-13 Thu 02:12:32:920.274 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bfb4c0 2025-02-13 Thu 02:12:32:920.289 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:920.330 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:920.356 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf9900 2025-02-13 Thu 02:12:32:920.363 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:920.372 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:920.673 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:920.692 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:920.721 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:920.738 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:920.748 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:920.841 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:32:920.877 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: SANDESH: Send FAILED: 1739412752920782 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.35:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:12:32:920.901 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300756383488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:12:32:924.447 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:924.491 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8fc0 2025-02-13 Thu 02:12:32:924.585 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfa240 2025-02-13 Thu 02:12:32:924.623 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bfab80 2025-02-13 Thu 02:12:32:924.658 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6180 2025-02-13 Thu 02:12:32:924.702 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf6ac0 2025-02-13 Thu 02:12:32:924.741 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7d40 2025-02-13 Thu 02:12:32:924.774 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf8680 2025-02-13 Thu 02:12:32:924.808 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1bf7400 2025-02-13 Thu 02:12:32:924.870 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf8fc0 2025-02-13 Thu 02:12:32:924.879 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:924.896 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:924.920 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bfa240 2025-02-13 Thu 02:12:32:924.926 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:924.955 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:924.965 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bfab80 2025-02-13 Thu 02:12:32:924.971 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:924.980 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:925.099 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf6ac0 2025-02-13 Thu 02:12:32:925.108 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:925.136 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:925.152 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf6180 2025-02-13 Thu 02:12:32:925.158 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:925.167 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:925.198 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf7d40 2025-02-13 Thu 02:12:32:925.205 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:925.214 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:925.234 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf8680 2025-02-13 Thu 02:12:32:925.240 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:925.249 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:925.266 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: RAC_Connect status: 0 0x1bf7400 2025-02-13 Thu 02:12:32:925.272 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:12:32:925.280 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:12:32:925.336 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:925.348 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:925.364 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:925.374 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:925.389 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:925.399 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:925.435 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:925.453 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:925.474 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:925.631 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:925.644 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:925.663 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:925.678 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:925.691 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:925.749 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:925.772 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:925.783 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:925.797 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:925.807 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:925.824 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:925.838 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:925.853 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:12:32:925.863 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:12:32:925.878 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:12:32:927.892 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:32:927.952 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: SANDESH: Send FAILED: 1739412752925731 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:12:32:927.973 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300672558848, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:12:37:402.010 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:37:402.125 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: SANDESH: Send FAILED: 1739412757401862 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 520 dequeues = 520 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 569 dequeues = 569 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739412751973925 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739412752927880 ] max_sm_queue_count = 6 ] 2025-02-13 Thu 02:12:37:402.140 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:12:37:402.153 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:37:402.312 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: SANDESH: Send FAILED: 1739412757401979 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 8 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 229 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 55 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 24 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 254, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 100, _iter59->first = sent _iter59->second = 420, ] ] ] 2025-02-13 Thu 02:12:37:402.350 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:12:42:390.298 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:12:42:390.407 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: SANDESH: Send FAILED: 1739412762390256 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-02-13 Thu 02:12:42:390.419 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300068579072, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-02-13 Thu 02:13:01:976.793 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-13 Thu 02:13:01:977.152 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Connect : Start Connect timer 10.0.0.35:8086 2025-02-13 Thu 02:13:01:977.267 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:13:01:977.301 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: SANDESH: Send FAILED: 1739412781976998 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:13:01:977.333 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:13:01:977.350 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:13:01:977.528 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: SANDESH: Send FAILED: 1739412781977252 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 8 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 229 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 57 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 24 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 254, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 103, _iter59->first = sent _iter59->second = 420, ] ] ] 2025-02-13 Thu 02:13:01:977.547 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300617996032, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:13:01:982.951 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.46:40940::10.0.0.35:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-13 Thu 02:13:01:982.986 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:40940::10.0.0.35:8086(-1) OnSessionEvent TCP Connected 2025-02-13 Thu 02:13:01:982.996 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300908451968, Pid 1]: 10.0.0.46:40940::10.0.0.35:8086(-1):EvTcpConnected 2025-02-13 Thu 02:13:01:991.786 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-13 Thu 02:13:01:991.842 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-13 Thu 02:13:01:991.902 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: ClientInit 2025-02-13 Thu 02:13:01:991.918 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2728-1.:contrail-query-engine:0:Database count 9 2025-02-13 Thu 02:13:01:992.020 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:13:01:992.030 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:13:01:992.057 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SANDESH: Send FAILED: 1739412781991890 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:13:01:992.066 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:13:01:992.076 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:13:01:992.084 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-13 Thu 02:13:01:992.096 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:13:01:992.103 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-13 Thu 02:13:01:992.261 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SANDESH: Send FAILED: 1739412781992008 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 9 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 229 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 58 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 25 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 255, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 105, _iter59->first = sent _iter59->second = 420, ] ] ] 2025-02-13 Thu 02:13:01:992.281 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-13 Thu 02:13:02:008.302 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: OnMessage control in state: ClientInit 2025-02-13 Thu 02:13:02:008.387 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-13 Thu 02:13:02:008.403 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-13 Thu 02:13:02:008.434 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Received Ctrl Message with size 0 2025-02-13 Thu 02:13:02:008.679 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 9 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3908297 calls = 416 average_bytes = 9394 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 520 dequeues = 520 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 569 dequeues = 569 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739412751973925 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739412752927880 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 21 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 176507 calls = 18 average_bytes = 9805 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 81 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 229 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 320 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 58 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 25 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 255, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 105, _iter59->first = sent _iter59->second = 420, ] ] max_sm_queue_count = 6 proxy seq 378 2025-02-13 Thu 02:13:02:008.715 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-13 Thu 02:13:02:008.747 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.35:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 55 2025-02-13 Thu 02:13:02:008.762 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-13 Thu 02:13:02:008.799 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2728-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 82 2025-02-13 Thu 02:13:02:008.811 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-02-13 Thu 02:13:02:008.827 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-13 Thu 02:13:02:008.876 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Established 2025-02-13 Thu 02:13:07:404.968 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 82 without seqno , total = 0 2025-02-13 Thu 02:13:08:111.957 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300752185088, Pid 1]: OnMessage control in state: Established 2025-02-13 Thu 02:13:08:112.035 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300752185088, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2025-02-13 Thu 02:13:08:112.048 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300752185088, Pid 1]: Established : EvSandeshMessageRecv 2025-02-13 Thu 02:13:08:112.098 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300752185088, Pid 1]: Received Ctrl Message with size 3 2025-02-13 Thu 02:13:08:112.115 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300752185088, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 381, total = 0 2025-02-13 Thu 02:13:08:112.128 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300752185088, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 57, total = 0 2025-02-13 Thu 02:13:08:112.139 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300752185088, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 82, total = 0 2025-02-13 Thu 02:13:08:112.148 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300752185088, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-13 Thu 02:14:07:405.678 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 83 without seqno , total = 0 2025-02-13 Thu 02:15:07:408.670 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300626392832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 84 without seqno , total = 0 2025-02-13 Thu 02:16:07:409.290 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300764780288, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 85 without seqno , total = 0 2025-02-13 Thu 02:16:50:208.351 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Processing scm::EvStop in state Established 2025-02-13 Thu 02:16:50:208.470 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Idle 2025-02-13 Thu 02:16:50:208.592 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-13 Thu 02:16:50:208.607 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-13 Thu 02:16:50:208.775 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: SANDESH: Send FAILED: 1739413010208579 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 9 pid = 1 http_port = 8091 start_time = 1739407826641653 collector_name = collector_ip = 10.0.0.35:8086 collector_list= [ [ (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 5913 calls = 10 average_bytes = 591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 4156820 calls = 440 average_bytes = 9447 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 46 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 86 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 229 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 338 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 59 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 25 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 255, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 107, _iter59->first = sent _iter59->second = 447, ] ] ] 2025-02-13 Thu 02:16:50:208.803 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 140300630591232, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:17:20:879.695 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: http-server-port 8091 2025-02-13 Thu 02:17:20:879.786 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Max-tasks 16 2025-02-13 Thu 02:17:20:879.795 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Max-slice 100 2025-02-13 Thu 02:17:20:879.802 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Collectors 10.0.0.46:8086 2025-02-13 Thu 02:17:20:879.810 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Collectors 10.0.0.35:8086 2025-02-13 Thu 02:17:20:879.817 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Collectors 10.0.0.12:8086 2025-02-13 Thu 02:17:20:879.870 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: SANDESH: ROLE : Generator 2025-02-13 Thu 02:17:20:879.880 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-13 Thu 02:17:20:879.886 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2728-1. 2025-02-13 Thu 02:17:20:879.892 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-13 Thu 02:17:20:879.898 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-13 Thu 02:17:20:879.904 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-13 Thu 02:17:20:879.911 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-13 Thu 02:17:20:881.626 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: HTTP Introspect Init 2025-02-13 Thu 02:17:20:881.743 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-13 Thu 02:17:20:881.812 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Sandesh Http Server Port: 8091 2025-02-13 Thu 02:17:20:881.853 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-13 Thu 02:17:20:881.862 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-13 Thu 02:17:20:881.953 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Idle 2025-02-13 Thu 02:17:20:882.130 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: SANDESH: No Client: 1739413040882044 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739413040881964 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 02:17:20:882.252 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-13 Thu 02:17:20:882.267 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-13 Thu 02:17:20:882.280 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Cassandra Servers: 10.0.0.46:9042 10.0.0.35:9042 10.0.0.12:9042 2025-02-13 Thu 02:17:20:882.408 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: Processing scm::EvStart in state Idle 2025-02-13 Thu 02:17:20:882.513 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: Disconnect 2025-02-13 Thu 02:17:20:882.614 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-13 Thu 02:17:20:882.829 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-02-13 Thu 02:17:20:882.906 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:17:20:882.950 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: SANDESH: Send FAILED: 1739413040882498 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-13 Thu 02:17:20:882.960 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:17:20:882.970 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:17:20:883.093 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: SANDESH: Send FAILED: 1739413040882601 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739413040881964 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 02:17:20:883.106 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:17:20:883.121 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:17:20:883.204 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: SANDESH: Send FAILED: 1739413040882670 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739413040881964 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 02:17:20:883.216 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:17:20:883.225 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:17:20:883.242 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: SANDESH: Send FAILED: 1739413040882691 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-13 Thu 02:17:20:883.253 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:17:20:883.262 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:17:20:883.344 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: SANDESH: Send FAILED: 1739413040882896 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739413040881964 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 02:17:20:883.355 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-13 Thu 02:17:20:929.725 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-13 Thu 02:17:20:935.894 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-13 Thu 02:17:20:935.953 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: SANDESH: Send FAILED: 1739413040934449 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-13 Thu 02:17:20:935.973 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-13 Thu 02:17:20:937.045 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.46:42466::10.0.0.12:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2025-02-13 Thu 02:17:20:937.066 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: 10.0.0.46:42466::10.0.0.12:8086(-1) OnSessionEvent TCP Connected 2025-02-13 Thu 02:17:20:937.075 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: 10.0.0.46:42466::10.0.0.12:8086(-1):EvTcpConnected 2025-02-13 Thu 02:17:20:937.115 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: RAC_Connect status: 0 0xc0f280 2025-02-13 Thu 02:17:20:937.125 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:17:20:937.139 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:17:20:937.153 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: RAC_Connect status: 0 0xc0f980 2025-02-13 Thu 02:17:20:937.159 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:17:20:937.168 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:17:20:937.179 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: RAC_Connect status: 0 0xc10190 2025-02-13 Thu 02:17:20:937.188 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:17:20:937.201 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:17:20:937.215 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: RAC_Connect status: 0 0xc10a10 2025-02-13 Thu 02:17:20:937.223 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:17:20:937.236 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:17:20:937.251 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: RAC_Connect status: 0 0xc11290 2025-02-13 Thu 02:17:20:937.260 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:17:20:937.274 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.46 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:17:20:937.299 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: RAC_Connect status: 0 0xc4f720 2025-02-13 Thu 02:17:20:937.308 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:17:20:937.332 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:17:20:937.346 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: RAC_Connect status: 0 0xc4ffa0 2025-02-13 Thu 02:17:20:937.355 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:17:20:937.369 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:17:20:937.383 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: RAC_Connect status: 0 0xc50820 2025-02-13 Thu 02:17:20:937.392 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:17:20:937.411 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:17:20:937.425 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: RAC_Connect status: 0 0xc510a0 2025-02-13 Thu 02:17:20:937.433 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:17:20:937.447 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:17:20:937.460 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: RAC_Connect status: 0 0xc4cca0 2025-02-13 Thu 02:17:20:937.469 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:17:20:937.479 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:17:20:937.489 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: RAC_Connect status: 0 0xc51920 2025-02-13 Thu 02:17:20:937.495 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:17:20:937.504 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:17:20:937.514 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: RAC_Connect status: 0 0xc4d520 2025-02-13 Thu 02:17:20:937.519 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:17:20:937.528 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:17:20:937.537 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: RAC_Connect status: 0 0xc4dda0 2025-02-13 Thu 02:17:20:937.543 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:17:20:937.551 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:17:20:937.565 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: RAC_Connect status: 0 0xc4e620 2025-02-13 Thu 02:17:20:937.571 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:17:20:937.580 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:17:20:937.590 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: RAC_Connect status: 0 0xc4eea0 2025-02-13 Thu 02:17:20:937.598 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: Connected to REDIS... 2025-02-13 Thu 02:17:20:937.611 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.35 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-13 Thu 02:17:20:937.903 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:17:20:937.923 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:17:20:937.974 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:17:20:937.991 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:17:20:938.002 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:17:20:938.075 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-13 Thu 02:17:20:938.089 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-13 Thu 02:17:20:938.117 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: ClientInit 2025-02-13 Thu 02:17:20:938.136 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2728-1.:contrail-query-engine:0:Database count 1 2025-02-13 Thu 02:17:20:938.239 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:17:20:938.249 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:17:20:938.274 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: SANDESH: Send FAILED: 1739413040937962 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:17:20:938.289 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:17:20:938.299 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:17:20:938.307 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:17:20:938.339 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: SANDESH: Send FAILED: 1739413040938111 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-13 Thu 02:17:20:938.347 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:17:20:938.356 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:17:20:938.364 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-13 Thu 02:17:20:938.389 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:17:20:938.396 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-13 Thu 02:17:20:938.528 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: SANDESH: Send FAILED: 1739413040938228 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739413040881964 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 23, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-13 Thu 02:17:20:938.544 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-13 Thu 02:17:20:938.673 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:17:20:938.686 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:17:20:938.703 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:17:20:938.718 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:17:20:938.732 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:17:20:938.742 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.46:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:17:20:938.784 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:17:20:938.835 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:17:20:938.854 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:17:20:938.871 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:17:20:938.896 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:17:20:938.907 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:17:20:938.921 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:17:20:938.931 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:17:20:938.944 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:17:20:938.954 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:17:20:938.967 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:17:20:938.977 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:17:20:938.990 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:17:20:938.999 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:17:20:939.017 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:17:20:939.028 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:17:20:939.041 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:17:20:939.051 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:17:20:939.064 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:17:20:939.074 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:17:20:939.087 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:17:20:939.097 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:17:20:939.110 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-13 Thu 02:17:20:939.120 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.35:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-13 Thu 02:17:20:939.144 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:17:20:939.170 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:17:20:939.186 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:17:20:939.202 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:17:20:939.217 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:17:20:939.246 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:17:20:939.279 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:17:20:939.296 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:17:20:939.315 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:17:20:939.337 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-13 Thu 02:17:20:939.401 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:17:20:939.412 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:17:20:939.436 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: SANDESH: Send FAILED: 1739413040938771 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.35:6379::Down.10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:17:20:939.444 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:17:20:939.454 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:17:20:939.462 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:17:20:939.483 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: SANDESH: Send FAILED: 1739413040939163 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = 10.0.0.35:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:17:20:939.490 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:17:20:939.499 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-13 Thu 02:17:20:939.506 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-13 Thu 02:17:20:939.524 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: SANDESH: Send FAILED: 1739413040939270 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-13 Thu 02:17:20:939.535 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-13 Thu 02:17:20:941.162 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: OnMessage control in state: ClientInit 2025-02-13 Thu 02:17:20:941.205 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-13 Thu 02:17:20:941.218 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-13 Thu 02:17:20:941.252 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Received Ctrl Message with size 0 2025-02-13 Thu 02:17:20:941.469 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2728-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739413040881964 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.46:8086, (*_iter40) = 10.0.0.35:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 23, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2025-02-13 Thu 02:17:20:941.502 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-13 Thu 02:17:20:941.539 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2728-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.46:9042, (*_iter6) = 10.0.0.35:9042, (*_iter6) = 10.0.0.12:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.46:6379, (*_iter6) = 10.0.0.35:6379, (*_iter6) = 10.0.0.12:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2025-02-13 Thu 02:17:20:941.555 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-13 Thu 02:17:20:941.567 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-13 Thu 02:17:20:941.579 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-13 Thu 02:17:20:941.614 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: Established 2025-02-13 Thu 02:18:20:883.403 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940570593024, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-13 Thu 02:19:20:884.324 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941386213120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-13 Thu 02:20:20:884.945 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941275215616, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-13 Thu 02:21:20:885.517 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940570593024, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-13 Thu 02:22:20:886.144 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940570593024, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-13 Thu 02:23:20:887.247 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941271017216, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-13 Thu 02:24:20:888.706 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940570593024, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-13 Thu 02:25:20:889.414 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-13 Thu 02:26:20:890.638 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941275215616, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-13 Thu 02:27:20:891.531 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941382014720, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-02-13 Thu 02:28:20:892.148 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941258422016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-02-13 Thu 02:29:20:893.642 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941398808320, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-02-13 Thu 02:30:20:894.246 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941266818816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-02-13 Thu 02:31:20:894.861 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-02-13 Thu 02:32:20:895.549 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940570593024, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-02-13 Thu 02:33:20:896.740 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941258422016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-02-13 Thu 02:33:54:356.629 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.46:8091::10.0.0.46:44004 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-13 Thu 02:34:20:897.310 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941382014720, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-02-13 Thu 02:34:23:447.907 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.46:8091::10.0.0.46:41700 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-13 Thu 02:34:44:229.018 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.46:8091::10.0.0.46:53770 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-13 Thu 02:34:57:053.489 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.46:8091::10.0.0.46:46626 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-13 Thu 02:35:20:899.702 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-02-13 Thu 02:36:20:901.841 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941266818816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-02-13 Thu 02:36:23:826.786 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.46:8091::10.0.0.46:50122 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-13 Thu 02:36:46:511.805 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.46 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 773 2025-02-13 Thu 02:36:46:512.116 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID 5e77759e-e9b3-11ef-be7a-00000a00002e src/contrail-analytics/contrail-query-engine/query.cc 1166 2025-02-13 Thu 02:36:46:512.164 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e :0:json query is: { end_time: 1739414206443616 , enqueue_time: 1739414206511079 , query_metadata: {"enqueue_time": 1739414206510893} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1739414188480058 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-13 Thu 02:36:46:512.354 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 880 2025-02-13 Thu 02:36:46:512.373 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 931 2025-02-13 Thu 02:36:46:512.540 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941254223616, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e :0:json query is: { end_time: 1739414206443616 , enqueue_time: 1739414206511079 , query_metadata: {"enqueue_time": 1739414206510893} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1739414188480058 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-13 Thu 02:36:46:518.595 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for 5e77759e-e9b3-11ef-be7a-00000a00002e , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 940 2025-02-13 Thu 02:36:46:701.846 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:46:701.888 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:46:701.899 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:46:701.909 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:46:701.918 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:46:701.926 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:46:701.934 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:46:701.942 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:46:701.951 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e UNION between tables of sizes 0 and 34 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:46:702.006 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e Resulting size of set 34 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:46:702.050 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e UNION between tables of sizes 0 and 10 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:46:702.091 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e Resulting size of set 10 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:46:702.111 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e UNION between tables of sizes 10 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:46:702.126 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e Resulting size of set 10 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:46:702.156 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e UNION between tables of sizes 10 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:46:702.097 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e :0:json query is: { end_time: 1739414206443616 , enqueue_time: 1739414206511079 , query_metadata: {"enqueue_time": 1739414206510893} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1739414188480058 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-13 Thu 02:36:46:702.171 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e Resulting size of set 10 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:46:702.200 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e UNION between tables of sizes 10 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:46:702.212 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e Resulting size of set 10 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:46:702.222 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e UNION between tables of sizes 10 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:46:702.233 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e Resulting size of set 10 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:46:702.449 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:702.531 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:702.663 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:702.745 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:702.858 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:702.942 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:703.036 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:703.134 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:703.212 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:703.356 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:703.395 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:703.550 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:703.565 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:703.739 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:703.812 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:703.940 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:704.023 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:704.119 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:704.222 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:704.297 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:704.418 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:704.475 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:704.632 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:704.784 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:704.933 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:705.105 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:705.259 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:705.420 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:705.569 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:705.718 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:705.865 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:706.014 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:706.161 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:46:706.176 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:46:706.186 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:46:706.195 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:46:706.204 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:46:706.220 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:46:706.229 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:46:706.237 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:46:706.246 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:46:706.254 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 5e77759e-e9b3-11ef-be7a-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:46:706.907 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:707.066 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:707.217 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:707.460 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:707.615 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:707.805 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:707.965 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:708.122 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:708.274 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:708.433 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:708.790 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:708.951 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: [SYS_INFO]: QEQueryLog: 5e77759e-e9b3-11ef-be7a-00000a00002e ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:46:709.235 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 44 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 520 2025-02-13 Thu 02:36:46:709.795 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 44 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 528 2025-02-13 Thu 02:36:46:709.956 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":6, "count":44} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 563 2025-02-13 Thu 02:36:46:710.902 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941382014720, Pid 1]: [SYS_INFO]: QELog: Finished: QID 5e77759e-e9b3-11ef-be7a-00000a00002e Table ObjectVNTable Time(ms) 198 RedisTime(ms) 0 MergeTime(ms) 0 Rows 44 EnQ-delay1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 644 2025-02-13 Thu 02:36:46:710.981 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941308790528, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-2728-1. table = ObjectVNTable query_stats= [ qid = 5e77759e-e9b3-11ef-be7a-00000a00002e where = [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-15013551:ctest-vn22-67883455", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 183,0, 183,0, 187,0, chunk_select_time = 0,6, 0,2, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 198 rows = 44 enq_delay = 1 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 636 2025-02-13 Thu 02:36:46:711.004 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941382014720, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 676 2025-02-13 Thu 02:36:49:679.417 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.46 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 773 2025-02-13 Thu 02:36:49:679.745 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID 605afb24-e9b3-11ef-a4bf-00000a00002e src/contrail-analytics/contrail-query-engine/query.cc 1166 2025-02-13 Thu 02:36:49:679.792 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QEQueryLog: 605afb24-e9b3-11ef-a4bf-00000a00002e :0:json query is: { end_time: 1739414209665415 , enqueue_time: 1739414209678968 , query_metadata: {"enqueue_time": 1739414209678840} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1739414188480058 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "1664be32-9074-4b6e-9354-a3b867117b57", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-13 Thu 02:36:49:680.001 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 880 2025-02-13 Thu 02:36:49:680.022 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 931 2025-02-13 Thu 02:36:49:680.074 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for 605afb24-e9b3-11ef-a4bf-00000a00002e , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 940 2025-02-13 Thu 02:36:49:680.193 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: [SYS_INFO]: QEQueryLog: 605afb24-e9b3-11ef-a4bf-00000a00002e :0:json query is: { end_time: 1739414209665415 , enqueue_time: 1739414209678968 , query_metadata: {"enqueue_time": 1739414209678840} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1739414188480058 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "1664be32-9074-4b6e-9354-a3b867117b57", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-13 Thu 02:36:49:719.674 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e UNION between tables of sizes 0 and 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:49:719.796 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:49:719.844 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e UNION between tables of sizes 5 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:49:719.859 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:49:719.869 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e UNION between tables of sizes 5 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:49:719.880 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:49:719.889 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e UNION between tables of sizes 5 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:49:719.899 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:49:719.908 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e UNION between tables of sizes 5 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:49:719.918 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:49:720.217 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QEQueryLog: 605afb24-e9b3-11ef-a4bf-00000a00002e ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 1664be32-9074-4b6e-9354-a3b867117b57 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:49:720.395 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QEQueryLog: 605afb24-e9b3-11ef-a4bf-00000a00002e ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 1664be32-9074-4b6e-9354-a3b867117b57 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:49:720.547 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QEQueryLog: 605afb24-e9b3-11ef-a4bf-00000a00002e ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 1664be32-9074-4b6e-9354-a3b867117b57 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:49:720.697 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QEQueryLog: 605afb24-e9b3-11ef-a4bf-00000a00002e ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 1664be32-9074-4b6e-9354-a3b867117b57 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:49:720.850 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QEQueryLog: 605afb24-e9b3-11ef-a4bf-00000a00002e ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 1664be32-9074-4b6e-9354-a3b867117b57 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-13 Thu 02:36:49:720.987 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:49:721.004 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:49:721.014 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:49:721.022 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:49:721.031 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:49:721.039 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:49:721.047 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:49:721.055 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:49:721.064 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:49:721.072 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:49:721.111 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941390411520, Pid 1]: [SYS_INFO]: QEQueryLog: 605afb24-e9b3-11ef-a4bf-00000a00002e :0:json query is: { end_time: 1739414209665415 , enqueue_time: 1739414209678968 , query_metadata: {"enqueue_time": 1739414209678840} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1739414188480058 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "1664be32-9074-4b6e-9354-a3b867117b57", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-13 Thu 02:36:49:722.425 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941266818816, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:49:722.462 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941266818816, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:49:722.472 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941266818816, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:49:722.482 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941266818816, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:49:722.490 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941266818816, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:49:722.499 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941266818816, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:49:722.507 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941266818816, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:49:722.515 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941266818816, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:49:722.524 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941266818816, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-13 Thu 02:36:49:722.532 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941266818816, Pid 1]: [SYS_INFO]: QELog: 605afb24-e9b3-11ef-a4bf-00000a00002e Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-13 Thu 02:36:49:722.897 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941266818816, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 520 2025-02-13 Thu 02:36:49:723.004 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941266818816, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 528 2025-02-13 Thu 02:36:49:723.058 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941266818816, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":5} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 563 2025-02-13 Thu 02:36:49:723.431 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941279414016, Pid 1]: [SYS_INFO]: QELog: Finished: QID 605afb24-e9b3-11ef-a4bf-00000a00002e Table ObjectVMTable Time(ms) 43 RedisTime(ms) 0 MergeTime(ms) 0 Rows 5 EnQ-delay0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 644 2025-02-13 Thu 02:36:49:723.480 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941271017216, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-2728-1. table = ObjectVMTable query_stats= [ qid = 605afb24-e9b3-11ef-a4bf-00000a00002e where = [[{"name": "ObjectId", "value": "1664be32-9074-4b6e-9354-a3b867117b57", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 40,0, 41,0, 38,0, chunk_select_time = 0,0, 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 43 rows = 5 enq_delay = 0 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 636 2025-02-13 Thu 02:36:49:723.494 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941279414016, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 676 2025-02-13 Thu 02:37:20:902.560 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139940566394624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-02-13 Thu 02:38:20:904.342 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941266818816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-02-13 Thu 02:39:20:904.999 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941266818816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-02-13 Thu 02:40:20:905.647 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941308790528, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-02-13 Thu 02:41:20:906.962 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941266818816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-02-13 Thu 02:42:20:907.591 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941266818816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-02-13 Thu 02:43:20:908.291 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941394609920, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2025-02-13 Thu 02:43:41:111.210 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.46:8091::10.0.0.46:40418 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-13 Thu 02:44:20:908.921 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941275215616, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2025-02-13 Thu 02:45:20:909.599 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941262620416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2025-02-13 Thu 02:46:00:711.022 UTC cn-jenkins-deploy-platform-ansible-os-2728-1 [Thread 139941546670208, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.46:8091::10.0.0.46:45936 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256