2025-02-17 Mon 00:52:53:288.445 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: http-server-port 8091 2025-02-17 Mon 00:52:53:288.542 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Max-tasks 16 2025-02-17 Mon 00:52:53:288.551 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Max-slice 100 2025-02-17 Mon 00:52:53:288.557 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Collectors 10.0.0.56:8086 2025-02-17 Mon 00:52:53:288.562 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Collectors 10.0.0.53:8086 2025-02-17 Mon 00:52:53:288.567 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Collectors 10.0.0.22:8086 2025-02-17 Mon 00:52:53:288.588 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: SANDESH: ROLE : Generator 2025-02-17 Mon 00:52:53:288.594 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-17 Mon 00:52:53:288.600 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2738-1. 2025-02-17 Mon 00:52:53:288.605 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-17 Mon 00:52:53:288.611 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-17 Mon 00:52:53:288.616 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-17 Mon 00:52:53:288.622 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-17 Mon 00:52:53:298.811 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: HTTP Introspect Init 2025-02-17 Mon 00:52:53:299.127 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-17 Mon 00:52:53:299.204 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Sandesh Http Server Port: 8091 2025-02-17 Mon 00:52:53:299.259 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-17 Mon 00:52:53:299.269 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-17 Mon 00:52:53:299.354 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Idle 2025-02-17 Mon 00:52:53:299.528 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: SANDESH: No Client: 1739753573299446 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 00:52:53:299.691 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-17 Mon 00:52:53:299.720 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-17 Mon 00:52:53:299.789 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Cassandra Servers: 10.0.0.56:9042 10.0.0.53:9042 10.0.0.22:9042 2025-02-17 Mon 00:52:53:299.928 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: Processing scm::EvStart in state Idle 2025-02-17 Mon 00:52:53:300.000 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: Disconnect 2025-02-17 Mon 00:52:53:300.074 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-17 Mon 00:52:53:300.297 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: Connect : Start Connect timer 10.0.0.22:8086 2025-02-17 Mon 00:52:53:300.371 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 00:52:53:300.394 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: SANDESH: Send FAILED: 1739753573299991 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-17 Mon 00:52:53:300.404 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 00:52:53:300.412 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 00:52:53:300.493 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: SANDESH: Send FAILED: 1739753573300063 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 00:52:53:300.509 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 00:52:53:300.519 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 00:52:53:300.587 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: SANDESH: Send FAILED: 1739753573300121 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 00:52:53:300.599 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 00:52:53:300.608 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 00:52:53:300.623 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: SANDESH: Send FAILED: 1739753573300142 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-17 Mon 00:52:53:300.630 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 00:52:53:300.637 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 00:52:53:300.740 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: SANDESH: Send FAILED: 1739753573300358 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 00:52:53:300.755 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 00:52:53:317.221 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-17 Mon 00:52:53:317.268 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:52:53:317.342 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:52:53:317.471 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 00:52:53:317.502 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: SANDESH: Send FAILED: 1739753573317307 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-17 Mon 00:52:53:317.512 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 00:52:58:340.925 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-17 Mon 00:52:58:340.977 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:52:58:341.024 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:03:353.007 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-17 Mon 00:53:03:353.061 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:03:353.092 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:08:355.238 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-17 Mon 00:53:08:355.314 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:08:355.348 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:13:388.595 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-17 Mon 00:53:13:388.661 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:13:388.694 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:18:476.792 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-17 Mon 00:53:18:476.846 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:18:476.883 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:23:507.562 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-17 Mon 00:53:23:507.635 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:23:507.661 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:28:531.328 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-17 Mon 00:53:28:531.411 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:28:531.441 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 7 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:33:571.918 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-17 Mon 00:53:33:571.976 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:33:572.001 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 8 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:38:614.342 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-17 Mon 00:53:38:614.401 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:38:614.424 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 9 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:43:729.605 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-17 Mon 00:53:43:729.652 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:43:729.685 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 10 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:48:763.381 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-17 Mon 00:53:48:763.434 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:48:763.458 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 11 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:53:815.847 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:53:815.907 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 12 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:58:859.536 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-17 Mon 00:54:03:868.951 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 00:54:03:869.002 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-17 Mon 00:54:03:869.038 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: -(-1):EvTcpConnectFail 2025-02-17 Mon 00:54:03:869.028 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: SANDESH: Send FAILED: 1739753643868834 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-17 Mon 00:54:03:869.116 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d2f620 2025-02-17 Mon 00:54:03:869.127 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:869.120 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 00:54:03:869.145 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:869.152 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-17 Mon 00:54:03:869.160 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d2fcd0 2025-02-17 Mon 00:54:03:869.171 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:869.161 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Connect : EvTcpConnectFail 2025-02-17 Mon 00:54:03:869.319 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-17 Mon 00:54:03:869.368 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Idle 2025-02-17 Mon 00:54:03:869.434 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 00:54:03:869.573 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: SANDESH: Send FAILED: 1739753643869268 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 41 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 42, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 00:54:03:869.598 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 00:54:03:869.609 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 00:54:03:869.629 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: SANDESH: Send FAILED: 1739753643869295 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-17 Mon 00:54:03:869.639 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 00:54:03:869.648 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 00:54:03:869.688 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 00:54:03:869.183 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:869.807 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: SANDESH: Send FAILED: 1739753643869423 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 41 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 43, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 00:54:03:869.817 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d29070 2025-02-17 Mon 00:54:03:869.842 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:869.830 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 00:54:03:869.862 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:869.874 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d300d0 2025-02-17 Mon 00:54:03:869.880 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:869.888 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:869.898 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d30840 2025-02-17 Mon 00:54:03:869.903 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:869.911 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:869.920 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d31180 2025-02-17 Mon 00:54:03:869.925 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:869.933 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:870.054 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d34900 2025-02-17 Mon 00:54:03:870.061 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:870.071 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:870.081 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d33fc0 2025-02-17 Mon 00:54:03:870.087 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:870.095 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:870.104 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d32400 2025-02-17 Mon 00:54:03:870.110 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:870.117 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:870.127 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d31ac0 2025-02-17 Mon 00:54:03:870.133 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:870.248 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:870.258 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d35240 2025-02-17 Mon 00:54:03:870.264 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:870.272 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:870.282 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d364c0 2025-02-17 Mon 00:54:03:870.288 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:870.300 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:870.313 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d32d40 2025-02-17 Mon 00:54:03:870.320 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:870.331 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:870.343 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d33680 2025-02-17 Mon 00:54:03:870.351 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:870.362 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:870.381 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d35b80 2025-02-17 Mon 00:54:03:870.389 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:870.399 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:870.590 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 00:54:03:870.644 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: SANDESH: Send FAILED: 1739753643870459 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739753643869367 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739753643869686 ] max_sm_queue_count = 5 ] 2025-02-17 Mon 00:54:03:870.653 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 00:54:03:870.662 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 00:54:03:870.787 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: SANDESH: Send FAILED: 1739753643870558 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 54 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 56, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 10, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 00:54:03:870.801 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 00:54:03:870.998 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:871.027 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:871.049 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:871.063 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:871.237 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:871.260 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:871.282 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:871.298 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:871.318 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:871.332 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:871.425 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:871.438 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 00:54:03:871.478 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:871.521 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:871.547 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:871.569 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:871.601 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:871.615 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:871.637 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:871.652 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:871.672 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:871.687 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:871.713 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:871.739 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:871.760 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:871.774 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:871.794 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:871.809 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:871.828 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:871.841 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:871.887 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:871.918 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:871.940 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:871.954 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:871.967 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:871.979 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:871.992 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:872.006 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:872.016 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:872.029 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:872.049 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:872.068 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:872.084 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:872.108 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:872.130 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:872.159 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:871.484 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: SANDESH: Send FAILED: 1739753643871386 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 00:54:03:872.371 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 00:54:03:872.385 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 00:54:03:872.410 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: SANDESH: Send FAILED: 1739753643871466 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.53:6379::Down.10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 00:54:03:872.418 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 00:54:03:872.426 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 00:54:03:872.453 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: SANDESH: Send FAILED: 1739753643871910 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 00:54:03:872.460 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 00:54:03:872.468 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 00:54:03:872.485 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: SANDESH: Send FAILED: 1739753643871934 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 00:54:03:872.492 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 00:54:04:869.573 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-17 Mon 00:54:04:869.886 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Connect : Start Connect timer 10.0.0.56:8086 2025-02-17 Mon 00:54:04:870.008 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 00:54:04:870.039 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: SANDESH: Send FAILED: 1739753644869666 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 00:54:04:870.048 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 00:54:04:870.057 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 00:54:04:870.193 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: SANDESH: Send FAILED: 1739753644869996 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 99 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 101, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 00:54:04:870.209 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 00:54:04:872.732 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.56:50938::10.0.0.56:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-17 Mon 00:54:04:872.767 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:50938::10.0.0.56:8086(-1) OnSessionEvent TCP Connected 2025-02-17 Mon 00:54:04:872.777 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:50938::10.0.0.56:8086(-1):EvTcpConnected 2025-02-17 Mon 00:54:04:873.590 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-17 Mon 00:54:04:873.628 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-17 Mon 00:54:04:873.680 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: ClientInit 2025-02-17 Mon 00:54:04:873.705 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2738-1.:contrail-query-engine:0:Database count 1 2025-02-17 Mon 00:54:04:873.842 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 00:54:04:873.858 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 00:54:04:873.890 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: SANDESH: Send FAILED: 1739753644873670 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 00:54:04:873.901 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 00:54:04:874.003 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 00:54:04:874.019 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-17 Mon 00:54:04:874.037 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 00:54:04:874.061 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-17 Mon 00:54:04:874.225 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: SANDESH: Send FAILED: 1739753644873826 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 99 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 102, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 00:54:04:874.251 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-17 Mon 00:54:04:879.486 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: OnMessage control in state: ClientInit 2025-02-17 Mon 00:54:04:879.554 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-17 Mon 00:54:04:879.570 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-17 Mon 00:54:04:879.605 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Received Ctrl Message with size 0 2025-02-17 Mon 00:54:04:879.859 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739753643869367 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739753643869686 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 99 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 102, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 5 proxy seq 10 2025-02-17 Mon 00:54:04:879.911 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-17 Mon 00:54:04:879.949 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2025-02-17 Mon 00:54:04:879.972 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-17 Mon 00:54:04:879.983 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-17 Mon 00:54:04:879.994 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-17 Mon 00:54:04:880.043 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: Established 2025-02-17 Mon 00:54:14:478.757 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: OnMessage control in state: Established 2025-02-17 Mon 00:54:14:478.822 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2025-02-17 Mon 00:54:14:478.833 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Established : EvSandeshMessageRecv 2025-02-17 Mon 00:54:14:478.876 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Received Ctrl Message with size 3 2025-02-17 Mon 00:54:14:478.894 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 11, total = 0 2025-02-17 Mon 00:54:14:478.907 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 13, total = 0 2025-02-17 Mon 00:54:14:488.378 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2025-02-17 Mon 00:54:14:488.401 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-17 Mon 00:54:33:873.234 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-17 Mon 00:55:33:876.328 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095198529280, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-17 Mon 00:56:33:876.910 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-17 Mon 00:57:25:142.284 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.56:8091::10.0.0.56:55972 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-02-17 Mon 00:57:33:877.835 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095198529280, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-17 Mon 00:57:36:595.032 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.56:8091::10.0.0.56:56216 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-02-17 Mon 00:58:33:878.999 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095113910016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-17 Mon 00:59:33:880.106 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-17 Mon 01:00:33:881.069 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095092926208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-17 Mon 01:01:33:882.660 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095092926208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-17 Mon 01:02:33:884.051 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095080335104, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-17 Mon 01:03:33:885.291 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095113910016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-02-17 Mon 01:04:33:887.569 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-02-17 Mon 01:05:33:889.152 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095113910016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-02-17 Mon 01:06:33:889.762 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-02-17 Mon 01:07:33:891.026 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-02-17 Mon 01:08:33:893.536 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-02-17 Mon 01:09:33:896.056 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-02-17 Mon 01:10:33:897.779 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095198529280, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-02-17 Mon 01:11:33:898.344 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095113910016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-02-17 Mon 01:12:33:898.979 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-02-17 Mon 01:13:33:899.624 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-02-17 Mon 01:14:33:900.273 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095198529280, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-02-17 Mon 01:15:33:901.351 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-02-17 Mon 01:16:33:902.941 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095198529280, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-02-17 Mon 01:17:33:904.025 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095092926208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-02-17 Mon 01:18:33:905.087 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095113910016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-02-17 Mon 01:19:33:905.653 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2025-02-17 Mon 01:20:33:906.183 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140094585435904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2025-02-17 Mon 01:21:33:906.724 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140094581237504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2025-02-17 Mon 01:22:33:907.227 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095113910016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0 2025-02-17 Mon 01:23:33:907.735 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140094585435904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 30 without seqno , total = 0 2025-02-17 Mon 01:24:33:908.307 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095092926208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 31 without seqno , total = 0 2025-02-17 Mon 01:25:33:908.884 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095092926208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 32 without seqno , total = 0 2025-02-17 Mon 01:26:33:909.470 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 33 without seqno , total = 0 2025-02-17 Mon 01:27:33:910.018 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140094585435904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 34 without seqno , total = 0 2025-02-17 Mon 01:28:33:910.563 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 35 without seqno , total = 0 2025-02-17 Mon 01:29:33:911.742 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095198529280, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 36 without seqno , total = 0 2025-02-17 Mon 01:30:33:912.339 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 37 without seqno , total = 0 2025-02-17 Mon 01:31:33:912.958 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095194330880, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 38 without seqno , total = 0 2025-02-17 Mon 01:32:33:913.618 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 39 without seqno , total = 0 2025-02-17 Mon 01:33:33:914.177 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140094585435904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 40 without seqno , total = 0 2025-02-17 Mon 01:34:33:914.770 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 41 without seqno , total = 0 2025-02-17 Mon 01:35:33:915.350 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095198529280, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 42 without seqno , total = 0 2025-02-17 Mon 01:36:33:915.910 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 43 without seqno , total = 0 2025-02-17 Mon 01:37:33:916.422 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140094585435904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 44 without seqno , total = 0 2025-02-17 Mon 01:38:33:917.050 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 45 without seqno , total = 0 2025-02-17 Mon 01:39:33:917.544 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 46 without seqno , total = 0 2025-02-17 Mon 01:40:33:918.194 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095194330880, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 47 without seqno , total = 0 2025-02-17 Mon 01:41:33:918.720 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095080335104, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 48 without seqno , total = 0 2025-02-17 Mon 01:42:33:919.261 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095092926208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 49 without seqno , total = 0 2025-02-17 Mon 01:43:33:919.885 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140094585435904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 50 without seqno , total = 0 2025-02-17 Mon 01:44:33:920.435 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 51 without seqno , total = 0 2025-02-17 Mon 01:45:33:921.162 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095113910016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 52 without seqno , total = 0 2025-02-17 Mon 01:46:33:921.738 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095113910016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 53 without seqno , total = 0 2025-02-17 Mon 01:47:33:922.627 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095198529280, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 54 without seqno , total = 0 2025-02-17 Mon 01:48:33:923.285 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095198529280, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 55 without seqno , total = 0 2025-02-17 Mon 01:49:33:923.824 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095084533504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 56 without seqno , total = 0 2025-02-17 Mon 01:50:33:924.468 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 57 without seqno , total = 0 2025-02-17 Mon 01:51:33:925.031 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095113910016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 58 without seqno , total = 0 2025-02-17 Mon 01:52:33:926.725 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 59 without seqno , total = 0 2025-02-17 Mon 01:53:23:830.345 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d33680 2025-02-17 Mon 01:53:23:830.399 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:53:23:830.440 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d32400 2025-02-17 Mon 01:53:23:830.452 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:53:23:830.472 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d31ac0 2025-02-17 Mon 01:53:23:830.483 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:53:23:830.501 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d32d40 2025-02-17 Mon 01:53:23:830.511 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:53:23:830.530 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-17 Mon 01:53:23:830.588 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d31180 2025-02-17 Mon 01:53:23:830.597 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:53:23:831.344 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:53:23:831.532 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:53:23:834.658 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:53:23:834.687 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:53:23:834.704 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:53:24:007.119 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d35b80 2025-02-17 Mon 01:53:24:007.196 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:53:24:007.357 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d34900 2025-02-17 Mon 01:53:24:007.377 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:53:24:007.407 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d364c0 2025-02-17 Mon 01:53:24:007.418 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:53:24:007.441 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-17 Mon 01:53:24:007.452 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d33fc0 2025-02-17 Mon 01:53:24:007.464 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:53:24:007.490 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d35240 2025-02-17 Mon 01:53:24:007.503 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:53:24:007.785 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:53:24:008.107 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:53:24:008.136 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:53:24:008.161 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:53:24:008.176 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:53:28:831.444 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:53:28:831.905 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:53:28:834.671 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:53:28:834.830 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:53:28:834.876 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:53:28:834.912 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:53:28:835.213 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:53:28:835.279 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:53:28:835.297 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:53:28:835.311 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:53:29:007.858 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:53:29:008.159 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:53:29:008.221 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:53:29:008.261 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:53:29:008.292 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:53:29:008.550 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:53:29:008.644 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:53:29:008.664 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:53:29:008.678 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:53:29:008.692 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:53:33:831.988 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:53:33:832.688 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:53:33:835.281 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:53:33:835.505 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:53:33:835.564 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:53:33:835.599 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:53:33:835.942 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:53:33:836.045 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:53:33:836.079 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:53:33:836.104 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:53:33:928.032 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095194330880, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 60 without seqno , total = 0 2025-02-17 Mon 01:53:34:008.678 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:53:34:008.983 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:53:34:009.076 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:53:34:009.111 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:53:34:009.162 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:53:34:009.459 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:53:34:009.518 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:53:34:009.534 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:53:34:009.548 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:53:34:009.720 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:53:38:832.808 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:53:38:833.570 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:53:38:836.017 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:53:38:836.159 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:53:38:836.218 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:53:38:836.259 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:53:38:836.545 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:53:38:836.576 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:53:38:836.642 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:53:38:842.445 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:53:39:010.430 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:53:39:010.630 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:53:39:010.670 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:53:39:010.697 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:53:39:010.721 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:53:39:011.572 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:53:39:011.615 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:53:39:011.630 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:53:39:011.644 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:53:39:011.657 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:53:43:833.667 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:53:43:834.329 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d33680 2025-02-17 Mon 01:53:43:834.351 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:53:43:834.376 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:53:43:834.799 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:53:43:834.820 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:53:43:834.848 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:53:43:836.603 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:53:43:836.718 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:53:43:836.811 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:53:43:837.105 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d32d40 2025-02-17 Mon 01:53:43:837.122 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:53:43:837.142 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:53:43:837.209 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d31180 2025-02-17 Mon 01:53:43:837.216 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:53:43:837.225 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:53:43:837.234 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d31ac0 2025-02-17 Mon 01:53:43:837.239 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:53:43:837.247 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:53:43:837.656 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:53:43:837.676 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:53:43:837.690 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:53:43:837.699 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:53:43:837.713 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:53:43:837.722 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:53:43:837.819 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:53:43:837.837 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:53:43:837.852 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:53:43:842.568 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:53:43:843.200 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d32400 2025-02-17 Mon 01:53:43:843.220 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:53:43:843.241 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:53:43:843.637 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:53:43:843.931 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:53:43:843.960 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:53:44:011.656 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:53:44:012.036 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:53:44:012.124 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:53:44:012.166 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:53:44:012.225 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:53:44:012.514 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d34900 2025-02-17 Mon 01:53:44:012.531 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:53:44:012.556 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:53:44:012.569 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d35b80 2025-02-17 Mon 01:53:44:012.575 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:53:44:012.584 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:53:44:012.612 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d35240 2025-02-17 Mon 01:53:44:012.618 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:53:44:012.626 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:53:44:012.666 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d33fc0 2025-02-17 Mon 01:53:44:012.675 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:53:44:012.687 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:53:44:012.699 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d364c0 2025-02-17 Mon 01:53:44:012.707 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:53:44:012.719 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:53:44:013.054 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:53:44:013.078 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:53:44:013.114 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:53:44:013.139 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:53:44:013.154 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:53:44:013.173 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:53:44:013.187 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:53:44:013.283 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:53:44:013.305 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:53:44:013.323 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:53:44:013.333 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:53:44:013.348 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:53:44:013.357 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:53:44:013.373 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:53:44:013.389 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:54:33:929.186 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 61 without seqno , total = 0 2025-02-17 Mon 01:55:33:930.180 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 62 without seqno , total = 0 2025-02-17 Mon 01:56:24:306.093 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d33680 2025-02-17 Mon 01:56:24:306.218 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:306.258 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d32400 2025-02-17 Mon 01:56:24:306.270 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:306.293 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d32d40 2025-02-17 Mon 01:56:24:306.302 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:306.322 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d31ac0 2025-02-17 Mon 01:56:24:306.332 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:306.353 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-17 Mon 01:56:24:306.360 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d31180 2025-02-17 Mon 01:56:24:306.369 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:310.568 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:56:24:310.623 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:56:24:310.640 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:56:24:310.659 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:56:24:310.683 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:56:24:310.706 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d30840 2025-02-17 Mon 01:56:24:310.733 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:310.785 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d300d0 2025-02-17 Mon 01:56:24:310.798 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:310.828 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d29070 2025-02-17 Mon 01:56:24:310.838 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:310.870 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d2fcd0 2025-02-17 Mon 01:56:24:310.880 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:310.906 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-17 Mon 01:56:24:310.913 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d2f620 2025-02-17 Mon 01:56:24:310.922 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:312.248 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:56:24:312.277 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:56:24:312.291 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:56:24:312.303 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:56:24:312.315 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:56:24:324.739 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:50938::10.0.0.56:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-17 Mon 01:56:24:324.784 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:50938::10.0.0.56:8086(-1):EvTcpClose 2025-02-17 Mon 01:56:24:325.532 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Processing scm::EvTcpClose in state Established 2025-02-17 Mon 01:56:24:325.554 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Established : EvTcpClose 2025-02-17 Mon 01:56:24:325.815 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Connect : Start Connect timer 10.0.0.53:8086 2025-02-17 Mon 01:56:24:325.944 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:24:326.099 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: SANDESH: Send FAILED: 1739757384325649 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = cn-jenkins-deploy-platform-ansible-os-2738-1. collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3036004 calls = 322 average_bytes = 9428 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 163 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 168, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 323, ] ] ] 2025-02-17 Mon 01:56:24:326.120 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:24:326.132 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:24:326.156 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: SANDESH: Send FAILED: 1739757384325679 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.56:6379::Down.10.0.0.53:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 01:56:24:326.163 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:24:326.172 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2025-02-17 Mon 01:56:24:326.184 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:24:326.311 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: SANDESH: Send FAILED: 1739757384325923 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3036004 calls = 322 average_bytes = 9428 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 163 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 168, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 323, ] ] ] 2025-02-17 Mon 01:56:24:326.326 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:24:333.805 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.56:47446::10.0.0.53:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-17 Mon 01:56:24:333.847 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:47446::10.0.0.53:8086(-1) OnSessionEvent TCP Connected 2025-02-17 Mon 01:56:24:333.859 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:47446::10.0.0.53:8086(-1):EvTcpConnected 2025-02-17 Mon 01:56:24:334.538 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-17 Mon 01:56:24:334.558 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-17 Mon 01:56:24:334.602 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: ClientInit 2025-02-17 Mon 01:56:24:334.622 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2738-1.:contrail-query-engine:0:Database count 2 2025-02-17 Mon 01:56:24:334.718 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:24:334.728 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 01:56:24:334.754 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: SANDESH: Send FAILED: 1739757384334593 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.53:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.56:6379::Down.10.0.0.53:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 01:56:24:334.779 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 01:56:24:334.789 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:24:334.796 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-17 Mon 01:56:24:334.807 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:24:334.814 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-17 Mon 01:56:24:334.951 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: SANDESH: Send FAILED: 1739757384334708 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3036004 calls = 322 average_bytes = 9428 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 163 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 169, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 323, ] ] ] 2025-02-17 Mon 01:56:24:334.967 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-17 Mon 01:56:24:357.899 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:47446::10.0.0.53:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-17 Mon 01:56:24:357.936 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:47446::10.0.0.53:8086(-1):EvTcpClose 2025-02-17 Mon 01:56:24:358.639 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-17 Mon 01:56:24:358.658 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: ClientInit : EvTcpClose 2025-02-17 Mon 01:56:24:358.811 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Return to idle with 1000 2025-02-17 Mon 01:56:24:358.822 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Idle 2025-02-17 Mon 01:56:24:358.881 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:24:359.055 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: SANDESH: Send FAILED: 1739757384358744 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3037577 calls = 323 average_bytes = 9404 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 163 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 169, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 25, _iter59->first = sent _iter59->second = 324, ] ] ] 2025-02-17 Mon 01:56:24:359.077 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:24:359.087 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:24:359.110 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: SANDESH: Send FAILED: 1739757384358787 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.53:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.56:6379::Down.10.0.0.53:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 01:56:24:359.120 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:24:359.130 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 01:56:24:359.139 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:24:359.268 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: SANDESH: Send FAILED: 1739757384358872 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3037577 calls = 323 average_bytes = 9404 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 163 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 169, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 25, _iter59->first = sent _iter59->second = 324, ] ] ] 2025-02-17 Mon 01:56:24:359.280 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:24:437.658 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-17 Mon 01:56:24:442.517 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d33fc0 2025-02-17 Mon 01:56:24:442.575 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:442.627 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d35b80 2025-02-17 Mon 01:56:24:442.639 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:442.665 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d364c0 2025-02-17 Mon 01:56:24:442.675 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:442.695 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d35240 2025-02-17 Mon 01:56:24:442.704 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:442.723 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Disconnect status: -1 0x1d34900 2025-02-17 Mon 01:56:24:442.732 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:442.883 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:24:442.933 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: SANDESH: Send FAILED: 1739757384442796 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.53:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:24:442.944 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:24:448.528 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:56:24:448.587 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:56:24:448.603 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:56:24:448.633 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:56:24:448.648 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:56:25:359.552 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-17 Mon 01:56:25:359.837 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Connect : Start Connect timer 10.0.0.22:8086 2025-02-17 Mon 01:56:25:359.996 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:25:360.041 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SANDESH: Send FAILED: 1739757385359653 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:25:360.054 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:25:360.064 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:25:360.217 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SANDESH: Send FAILED: 1739757385359970 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3037577 calls = 323 average_bytes = 9404 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 175, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 29, _iter59->first = sent _iter59->second = 324, ] ] ] 2025-02-17 Mon 01:56:25:360.250 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:25:364.314 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.56:54152::10.0.0.22:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-17 Mon 01:56:25:364.346 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:54152::10.0.0.22:8086(-1) OnSessionEvent TCP Connected 2025-02-17 Mon 01:56:25:364.354 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:54152::10.0.0.22:8086(-1):EvTcpConnected 2025-02-17 Mon 01:56:25:366.955 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-17 Mon 01:56:25:366.982 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-17 Mon 01:56:25:367.023 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: ClientInit 2025-02-17 Mon 01:56:25:367.042 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2738-1.:contrail-query-engine:0:Database count 3 2025-02-17 Mon 01:56:25:367.141 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:25:367.150 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 01:56:25:367.212 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: SANDESH: Send FAILED: 1739757385367015 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:25:367.223 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 01:56:25:367.233 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:25:367.240 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-17 Mon 01:56:25:367.252 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:25:367.258 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-17 Mon 01:56:25:367.396 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: SANDESH: Send FAILED: 1739757385367130 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3037577 calls = 323 average_bytes = 9404 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 176, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 31, _iter59->first = sent _iter59->second = 324, ] ] ] 2025-02-17 Mon 01:56:25:367.421 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-17 Mon 01:56:25:370.748 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:54152::10.0.0.22:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-17 Mon 01:56:25:370.779 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:54152::10.0.0.22:8086(-1):EvTcpClose 2025-02-17 Mon 01:56:25:373.276 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-17 Mon 01:56:25:373.297 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: ClientInit : EvTcpClose 2025-02-17 Mon 01:56:25:373.433 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Return to idle with 1000 2025-02-17 Mon 01:56:25:373.442 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Idle 2025-02-17 Mon 01:56:25:373.499 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:25:373.733 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SANDESH: Send FAILED: 1739757385373376 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3039150 calls = 324 average_bytes = 9380 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 176, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 33, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-17 Mon 01:56:25:373.782 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:25:373.795 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:25:373.817 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SANDESH: Send FAILED: 1739757385373405 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:25:373.825 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:25:373.975 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 01:56:25:373.989 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:25:374.120 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SANDESH: Send FAILED: 1739757385373490 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3039150 calls = 324 average_bytes = 9380 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 176, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 33, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-17 Mon 01:56:25:374.138 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:26:375.951 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-17 Mon 01:56:26:382.910 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Connect : Start Connect timer 10.0.0.56:8086 2025-02-17 Mon 01:56:26:383.055 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:26:383.090 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: SANDESH: Send FAILED: 1739757386376030 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:26:383.099 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:26:383.110 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:26:383.257 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: SANDESH: Send FAILED: 1739757386383023 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3039150 calls = 324 average_bytes = 9380 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 176, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 36, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-17 Mon 01:56:26:383.281 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:26:385.425 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.56:57604::10.0.0.56:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-17 Mon 01:56:26:385.462 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:57604::10.0.0.56:8086(-1) OnSessionEvent TCP Connected 2025-02-17 Mon 01:56:26:385.475 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:57604::10.0.0.56:8086(-1):EvTcpConnected 2025-02-17 Mon 01:56:26:388.146 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-17 Mon 01:56:26:388.167 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-17 Mon 01:56:26:388.210 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: ClientInit 2025-02-17 Mon 01:56:26:388.229 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2738-1.:contrail-query-engine:0:Database count 4 2025-02-17 Mon 01:56:26:388.320 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:26:388.338 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 01:56:26:388.364 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SANDESH: Send FAILED: 1739757386388202 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:26:388.375 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 01:56:26:388.394 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:26:388.404 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-17 Mon 01:56:26:388.420 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:26:388.429 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-17 Mon 01:56:26:388.616 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SANDESH: Send FAILED: 1739757386388309 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3039150 calls = 324 average_bytes = 9380 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 177, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 38, _iter59->first = sent _iter59->second = 325, ] ] ] 2025-02-17 Mon 01:56:26:388.639 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-17 Mon 01:56:26:391.104 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:57604::10.0.0.56:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-17 Mon 01:56:26:391.125 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:57604::10.0.0.56:8086(-1):EvTcpClose 2025-02-17 Mon 01:56:26:393.748 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-17 Mon 01:56:26:393.773 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: ClientInit : EvTcpClose 2025-02-17 Mon 01:56:26:393.929 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Return to idle with 1000 2025-02-17 Mon 01:56:26:393.943 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Idle 2025-02-17 Mon 01:56:26:394.022 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:26:394.171 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: SANDESH: Send FAILED: 1739757386393855 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3040723 calls = 325 average_bytes = 9356 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 177, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 326, ] ] ] 2025-02-17 Mon 01:56:26:394.195 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:26:394.205 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:26:394.227 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: SANDESH: Send FAILED: 1739757386393891 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:26:394.234 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:26:394.243 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 01:56:26:394.252 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:26:394.377 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: SANDESH: Send FAILED: 1739757386394010 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3040723 calls = 325 average_bytes = 9356 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 177, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 326, ] ] ] 2025-02-17 Mon 01:56:26:394.394 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:27:396.661 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-17 Mon 01:56:27:396.997 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Connect : Start Connect timer 10.0.0.53:8086 2025-02-17 Mon 01:56:27:397.120 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:27:397.160 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SANDESH: Send FAILED: 1739757387396750 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.53:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:27:397.172 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:27:397.182 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:27:397.324 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SANDESH: Send FAILED: 1739757387397101 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3040723 calls = 325 average_bytes = 9356 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 177, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 43, _iter59->first = sent _iter59->second = 326, ] ] ] 2025-02-17 Mon 01:56:27:397.350 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:27:399.810 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.56:47486::10.0.0.53:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-17 Mon 01:56:27:399.837 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:47486::10.0.0.53:8086(-1) OnSessionEvent TCP Connected 2025-02-17 Mon 01:56:27:399.845 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:47486::10.0.0.53:8086(-1):EvTcpConnected 2025-02-17 Mon 01:56:27:402.408 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-17 Mon 01:56:27:402.429 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-17 Mon 01:56:27:402.472 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: ClientInit 2025-02-17 Mon 01:56:27:402.485 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2738-1.:contrail-query-engine:0:Database count 5 2025-02-17 Mon 01:56:27:402.584 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:27:402.594 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 01:56:27:402.617 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: SANDESH: Send FAILED: 1739757387402464 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.53:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:27:402.625 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 01:56:27:402.634 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:27:402.641 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-17 Mon 01:56:27:402.652 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:27:402.668 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-17 Mon 01:56:27:402.830 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: SANDESH: Send FAILED: 1739757387402573 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3040723 calls = 325 average_bytes = 9356 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 178, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 45, _iter59->first = sent _iter59->second = 326, ] ] ] 2025-02-17 Mon 01:56:27:402.847 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-17 Mon 01:56:29:310.687 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:56:29:310.861 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:56:29:310.905 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:56:29:310.941 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:56:29:310.972 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:56:29:311.261 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:56:29:311.339 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:56:29:311.360 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:56:29:311.397 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:56:29:311.418 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:56:29:312.364 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:56:29:312.498 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:56:29:312.558 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:56:29:312.605 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:56:29:312.642 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:56:29:312.683 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:56:29:312.706 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:56:29:312.719 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:56:29:312.732 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:56:29:312.744 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:56:29:448.657 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:56:29:448.929 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:56:29:448.987 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:56:29:449.038 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:56:29:449.079 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:56:29:449.383 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:56:29:449.447 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:56:29:449.468 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:56:29:449.489 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:56:29:449.509 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:56:33:935.160 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 63 without seqno , total = 0 2025-02-17 Mon 01:56:33:935.386 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:33:935.407 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-17 Mon 01:56:33:935.563 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SANDESH: Send FAILED: 1739757393935011 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 374 dequeues = 374 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 397 dequeues = 397 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = ClientInit state_since = 1739757387402471 last_state = ClientInit last_event = scm::EvSandeshSend last_event_at = 1739757387402651 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 1 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 5 ] 2025-02-17 Mon 01:56:33:935.582 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-17 Mon 01:56:33:935.596 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:33:935.607 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-17 Mon 01:56:33:935.844 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SANDESH: Send FAILED: 1739757393935133 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3042296 calls = 326 average_bytes = 9332 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 178, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 47, _iter59->first = sent _iter59->second = 327, ] ] ] 2025-02-17 Mon 01:56:33:935.874 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-17 Mon 01:56:34:311.370 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:56:34:311.615 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:56:34:311.660 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:56:34:311.689 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:56:34:311.730 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:56:34:311.943 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:56:34:312.018 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:56:34:312.037 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:56:34:312.053 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:56:34:312.095 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:56:34:312.724 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:56:34:312.834 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:56:34:312.913 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:56:34:312.936 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:56:34:312.947 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:56:34:312.976 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:56:34:313.003 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:56:34:313.034 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:56:34:313.049 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:56:34:313.074 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:56:34:449.462 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:56:34:449.664 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:56:34:449.703 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:56:34:449.732 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:56:34:449.773 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:56:34:450.151 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:56:34:450.194 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:56:34:450.210 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:56:34:450.225 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:56:34:450.238 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:56:35:404.880 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: 10.0.0.53:8086 EvConnectTimerExpired in state ClientInit 2025-02-17 Mon 01:56:35:404.956 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Processing scm::EvConnectTimerExpired in state ClientInit 2025-02-17 Mon 01:56:35:404.966 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: ClientInit : EvConnectTimerExpired 2025-02-17 Mon 01:56:35:405.183 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Return to idle with 1000 2025-02-17 Mon 01:56:35:405.196 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Idle 2025-02-17 Mon 01:56:35:405.269 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:35:405.430 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: SANDESH: Send FAILED: 1739757395405050 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3042296 calls = 326 average_bytes = 9332 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 178, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 49, _iter59->first = sent _iter59->second = 327, ] ] ] 2025-02-17 Mon 01:56:35:405.459 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:35:405.472 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:35:405.502 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: SANDESH: Send FAILED: 1739757395405093 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.53:8086, ] ] status = Down description = ClientInit : EvConnectTimerExpired ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:35:405.513 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:35:405.522 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 01:56:35:405.531 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:35:405.659 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: SANDESH: Send FAILED: 1739757395405259 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3042296 calls = 326 average_bytes = 9332 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 178, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 49, _iter59->first = sent _iter59->second = 327, ] ] ] 2025-02-17 Mon 01:56:35:405.674 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:36:408.525 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-17 Mon 01:56:36:408.776 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Connect : Start Connect timer 10.0.0.22:8086 2025-02-17 Mon 01:56:36:408.902 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:36:408.936 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SANDESH: Send FAILED: 1739757396408612 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:36:408.945 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:36:408.954 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:36:409.098 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SANDESH: Send FAILED: 1739757396408884 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3042296 calls = 326 average_bytes = 9332 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 178, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 52, _iter59->first = sent _iter59->second = 327, ] ] ] 2025-02-17 Mon 01:56:36:409.122 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:36:414.217 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.56:52800::10.0.0.22:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-17 Mon 01:56:36:414.262 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:52800::10.0.0.22:8086(-1) OnSessionEvent TCP Connected 2025-02-17 Mon 01:56:36:414.270 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:52800::10.0.0.22:8086(-1):EvTcpConnected 2025-02-17 Mon 01:56:36:414.680 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-17 Mon 01:56:36:414.701 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-17 Mon 01:56:36:414.742 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: ClientInit 2025-02-17 Mon 01:56:36:414.772 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2738-1.:contrail-query-engine:0:Database count 6 2025-02-17 Mon 01:56:36:414.864 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:36:414.874 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 01:56:36:414.898 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: SANDESH: Send FAILED: 1739757396414734 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:36:414.916 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 01:56:36:414.926 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:36:414.933 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-17 Mon 01:56:36:414.945 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:36:414.951 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-17 Mon 01:56:36:415.090 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: SANDESH: Send FAILED: 1739757396414852 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3042296 calls = 326 average_bytes = 9332 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 179, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 54, _iter59->first = sent _iter59->second = 327, ] ] ] 2025-02-17 Mon 01:56:36:415.105 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095122306816, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-17 Mon 01:56:37:054.424 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.56:52800::10.0.0.22:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2025-02-17 Mon 01:56:37:054.487 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.56:52800::10.0.0.22:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-17 Mon 01:56:37:054.510 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:52800::10.0.0.22:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-17 Mon 01:56:37:054.518 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:52800::10.0.0.22:8086(-1):EvTcpClose 2025-02-17 Mon 01:56:37:054.604 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-17 Mon 01:56:37:054.639 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: ClientInit : EvTcpClose 2025-02-17 Mon 01:56:37:054.797 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Return to idle with 1000 2025-02-17 Mon 01:56:37:054.809 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Idle 2025-02-17 Mon 01:56:37:054.881 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:37:055.048 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: SANDESH: Send FAILED: 1739757397054726 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 181, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 56, _iter59->first = sent _iter59->second = 328, ] ] ] 2025-02-17 Mon 01:56:37:055.077 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:37:055.093 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:37:055.118 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: SANDESH: Send FAILED: 1739757397054774 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:37:055.130 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:37:055.144 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 01:56:37:055.178 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:37:055.320 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: SANDESH: Send FAILED: 1739757397054870 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 181, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 56, _iter59->first = sent _iter59->second = 328, ] ] ] 2025-02-17 Mon 01:56:37:055.343 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:38:054.914 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-17 Mon 01:56:38:055.171 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Connect : Start Connect timer 10.0.0.56:8086 2025-02-17 Mon 01:56:38:055.294 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:38:055.337 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SANDESH: Send FAILED: 1739757398055003 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:38:055.350 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:38:055.363 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:38:055.516 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SANDESH: Send FAILED: 1739757398055275 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 181, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 59, _iter59->first = sent _iter59->second = 328, ] ] ] 2025-02-17 Mon 01:56:38:055.542 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:38:055.950 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-17 Mon 01:56:38:055.961 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: -(-1):EvTcpConnectFail 2025-02-17 Mon 01:56:38:056.675 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-17 Mon 01:56:38:056.690 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Connect : EvTcpConnectFail 2025-02-17 Mon 01:56:38:056.830 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-17 Mon 01:56:38:056.878 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Idle 2025-02-17 Mon 01:56:38:056.941 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:38:057.080 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: SANDESH: Send FAILED: 1739757398056780 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 181, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 61, _iter59->first = sent _iter59->second = 328, ] ] ] 2025-02-17 Mon 01:56:38:057.102 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:38:057.112 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:38:057.133 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: SANDESH: Send FAILED: 1739757398056806 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:38:057.140 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:38:057.149 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 01:56:38:057.158 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:38:057.297 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: SANDESH: Send FAILED: 1739757398056931 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 182, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 61, _iter59->first = sent _iter59->second = 328, ] ] ] 2025-02-17 Mon 01:56:38:057.314 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:39:312.061 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:56:39:312.243 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:56:39:312.293 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:56:39:312.330 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:56:39:312.369 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:56:39:312.830 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:56:39:312.893 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:56:39:312.912 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:56:39:312.933 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:56:39:312.947 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:56:39:312.965 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:56:39:313.029 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:56:39:313.084 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:56:39:313.100 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:56:39:313.111 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:56:39:313.140 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:56:39:313.174 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:56:39:313.188 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:56:39:313.202 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:56:39:313.233 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:56:39:452.924 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:56:39:453.126 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:56:39:453.177 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:56:39:453.221 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:56:39:453.258 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:56:39:456.921 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:56:39:456.998 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:56:39:457.024 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:56:39:457.047 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:56:39:457.069 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:56:44:313.126 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:56:44:313.278 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:56:44:313.316 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:56:44:313.343 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:56:44:313.371 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:56:44:313.402 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:56:44:313.446 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:56:44:313.475 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:56:44:313.503 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:56:44:313.733 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:56:44:313.856 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:56:44:313.898 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:56:44:313.920 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:56:44:313.941 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:56:44:313.962 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:56:44:313.982 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:56:44:314.009 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:56:44:314.024 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:56:44:314.037 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:56:44:314.052 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:56:44:457.140 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:56:44:457.339 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:56:44:457.392 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:56:44:457.426 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:56:44:457.465 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:56:44:457.854 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:56:44:457.906 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:56:44:457.923 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:56:44:457.938 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:56:44:457.952 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:56:49:313.955 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:56:49:314.125 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:56:49:314.179 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:56:49:314.206 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:56:49:314.217 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:56:49:314.246 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:56:49:314.282 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:56:49:314.312 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:56:49:314.336 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:56:49:314.366 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:56:49:314.397 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:56:49:314.442 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:56:49:314.474 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:56:49:314.490 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:56:49:314.505 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:56:49:314.734 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:56:49:314.808 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:56:49:314.839 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:56:49:314.855 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:56:49:314.874 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:56:49:457.958 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:56:49:458.115 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:56:49:458.159 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:56:49:458.186 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:56:49:458.215 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:56:49:458.501 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:56:49:458.545 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:56:49:458.578 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:56:49:458.600 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:56:49:458.615 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:56:54:314.255 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:56:54:314.424 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:56:54:314.473 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:56:54:314.509 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:56:54:314.527 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:56:54:314.558 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:56:54:314.584 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:56:54:314.630 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:56:54:314.645 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:56:54:314.658 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:56:54:314.807 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:56:54:314.871 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:56:54:314.898 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:56:54:314.927 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:56:54:314.951 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:56:54:315.233 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:56:54:315.270 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:56:54:315.283 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:56:54:315.296 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:56:54:315.335 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:56:54:458.600 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:56:54:458.802 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:56:54:458.864 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:56:54:458.902 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:56:54:458.938 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:56:54:460.216 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:56:54:460.276 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:56:54:460.292 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:56:54:460.306 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:56:54:460.320 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:56:59:314.552 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:56:59:314.737 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:56:59:314.830 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:56:59:314.886 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:56:59:314.899 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:56:59:314.939 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:56:59:314.978 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:56:59:315.016 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:56:59:315.032 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:56:59:315.044 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:56:59:315.347 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:56:59:315.410 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:56:59:315.438 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:56:59:315.461 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:56:59:315.491 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:56:59:315.849 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:56:59:315.895 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:56:59:315.920 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:56:59:315.940 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:56:59:315.986 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:56:59:460.311 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:56:59:460.467 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:56:59:460.503 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:56:59:460.531 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:56:59:460.556 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:56:59:460.913 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:56:59:460.969 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:56:59:460.984 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:56:59:460.999 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:56:59:461.026 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:57:03:936.486 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:03:936.603 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SANDESH: Send FAILED: 1739757423936337 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 392 dequeues = 392 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 424 dequeues = 424 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739757398056876 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739757398057157 ] max_sm_queue_count = 5 ] 2025-02-17 Mon 01:57:03:936.618 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:57:03:936.629 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:03:936.825 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SANDESH: Send FAILED: 1739757423936445 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 182, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 64, _iter59->first = sent _iter59->second = 328, ] ] ] 2025-02-17 Mon 01:57:03:936.900 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:57:04:314.926 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:57:04:315.109 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:57:04:315.141 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:57:04:315.186 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:57:04:315.228 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:57:04:315.281 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:57:04:315.327 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:57:04:315.349 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:57:04:315.367 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:57:04:315.384 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:57:04:315.896 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:57:04:315.971 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:57:04:316.015 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:57:04:316.049 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:57:04:316.087 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:57:04:316.284 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:57:04:316.338 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:57:04:316.366 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:57:04:316.395 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:57:04:316.416 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:57:04:461.049 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:57:04:461.194 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:57:04:461.228 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:57:04:461.273 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:57:04:461.302 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:57:04:461.675 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:57:04:461.716 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:57:04:461.731 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:57:04:461.743 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:57:04:461.756 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:57:08:057.755 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-17 Mon 01:57:08:058.031 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Connect : Start Connect timer 10.0.0.53:8086 2025-02-17 Mon 01:57:08:058.135 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:57:08:058.176 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SANDESH: Send FAILED: 1739757428057860 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.53:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:57:08:058.189 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:57:08:058.203 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:57:08:058.450 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SANDESH: Send FAILED: 1739757428058119 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 182, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 66, _iter59->first = sent _iter59->second = 328, ] ] ] 2025-02-17 Mon 01:57:08:058.490 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:57:08:058.684 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-17 Mon 01:57:08:058.696 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: -(-1):EvTcpConnectFail 2025-02-17 Mon 01:57:08:058.754 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-17 Mon 01:57:08:058.811 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Connect : EvTcpConnectFail 2025-02-17 Mon 01:57:08:058.934 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-17 Mon 01:57:08:059.120 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Idle 2025-02-17 Mon 01:57:08:059.197 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:08:059.365 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SANDESH: Send FAILED: 1739757428058885 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 182, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 68, _iter59->first = sent _iter59->second = 328, ] ] ] 2025-02-17 Mon 01:57:08:059.393 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:57:08:059.404 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:08:059.427 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SANDESH: Send FAILED: 1739757428058913 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:57:08:059.434 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:57:08:059.443 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 01:57:08:059.468 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:08:059.599 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SANDESH: Send FAILED: 1739757428059185 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 183, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 68, _iter59->first = sent _iter59->second = 328, ] ] ] 2025-02-17 Mon 01:57:08:059.616 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:57:08:932.911 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095198529280, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:08:933.000 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095198529280, Pid 1]: SANDESH: Send FAILED: 1739757428932871 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-02-17 Mon 01:57:08:933.013 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095198529280, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-02-17 Mon 01:57:09:315.386 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:57:09:315.547 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:57:09:315.613 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:57:09:315.645 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:57:09:315.660 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:57:09:315.701 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:57:09:315.737 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:57:09:315.802 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:57:09:315.823 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:57:09:315.840 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:57:09:316.919 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:57:09:317.019 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:57:09:317.060 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:57:09:317.100 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:57:09:317.141 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:57:09:317.366 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:57:09:317.404 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:57:09:317.424 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:57:09:319.251 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:57:09:319.300 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:57:09:461.748 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:57:09:461.925 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:57:09:461.959 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:57:09:461.986 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:57:09:462.015 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:57:09:462.421 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:57:09:462.453 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:57:09:462.468 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:57:09:462.482 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:57:09:462.495 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:57:14:315.723 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:57:14:315.919 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:57:14:315.989 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:57:14:316.017 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:57:14:316.028 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:57:14:316.060 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:57:14:316.087 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:57:14:316.119 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:57:14:316.134 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:57:14:316.146 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:57:14:317.429 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:57:14:317.551 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:57:14:317.596 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:57:14:318.029 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:57:14:318.081 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:57:14:318.103 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:57:14:319.304 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:57:14:319.396 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:57:14:319.691 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:57:14:319.714 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:57:14:465.376 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:57:14:465.537 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:57:14:465.576 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:57:14:465.602 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:57:14:465.631 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:57:14:465.936 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:57:14:466.063 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:57:14:466.080 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:57:14:466.108 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:57:14:466.122 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:57:19:316.069 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:57:19:316.229 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:57:19:316.285 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:57:19:316.312 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:57:19:316.323 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:57:19:316.351 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:57:19:316.379 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:57:19:316.411 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:57:19:316.425 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:57:19:316.438 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:57:19:318.102 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:57:19:318.216 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:57:19:318.247 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:57:19:318.608 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:57:19:319.060 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:57:19:319.091 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:57:19:319.732 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:57:19:319.822 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:57:19:320.228 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:57:19:320.298 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:57:19:466.024 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:57:19:466.215 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:57:19:466.283 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:57:19:466.329 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:57:19:466.385 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:57:19:466.667 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:57:19:466.716 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:57:19:466.739 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:57:19:466.773 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:57:19:466.792 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:57:24:316.350 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:57:24:316.491 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:57:24:316.546 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:57:24:316.572 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:57:24:316.583 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:57:24:316.611 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:57:24:316.640 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:57:24:316.673 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:57:24:316.688 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:57:24:316.700 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:57:24:318.665 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:57:24:319.155 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:57:24:319.190 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:57:24:319.239 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:57:24:319.500 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:57:24:319.534 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:57:24:320.285 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:57:24:320.356 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:57:24:320.682 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:57:24:320.725 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:57:24:466.751 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:57:24:466.934 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:57:24:466.972 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:57:24:466.999 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:57:24:467.027 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:57:24:467.448 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:57:24:467.527 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:57:24:467.544 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:57:24:467.559 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:57:24:467.573 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:57:29:316.628 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:57:29:316.807 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:57:29:316.876 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:57:29:316.907 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:57:29:316.918 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:57:29:316.950 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:57:29:316.977 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:57:29:317.010 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:57:29:317.025 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:57:29:317.037 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:57:29:319.234 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:57:29:319.549 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:57:29:319.604 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:57:29:319.725 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:57:29:319.935 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:57:29:320.003 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:57:29:320.744 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:57:29:320.833 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:57:29:321.082 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:57:29:321.131 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:57:29:467.547 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:57:29:467.798 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:57:29:467.860 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:57:29:467.902 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:57:29:467.938 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:57:29:468.222 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:57:29:468.290 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:57:29:468.305 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:57:29:468.319 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:57:29:468.332 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:57:33:937.254 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 64 without seqno , total = 0 2025-02-17 Mon 01:57:33:937.310 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:33:937.375 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SANDESH: Send FAILED: 1739757453937139 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 400 dequeues = 400 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 435 dequeues = 435 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739757428059117 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739757428932896 ] max_sm_queue_count = 5 ] 2025-02-17 Mon 01:57:33:937.399 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:57:33:937.408 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:33:937.544 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SANDESH: Send FAILED: 1739757453937238 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 183, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 72, _iter59->first = sent _iter59->second = 328, ] ] ] 2025-02-17 Mon 01:57:33:937.558 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:57:34:316.945 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:57:34:317.093 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:57:34:317.117 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:57:34:317.150 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:57:34:317.179 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:57:34:317.217 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:57:34:317.265 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:57:34:317.287 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:57:34:317.303 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:57:34:317.318 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:57:34:319.799 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:57:34:320.009 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:57:34:320.089 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:57:34:320.403 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:57:34:320.433 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:57:34:320.448 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:57:34:321.147 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:57:34:321.223 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:57:34:321.478 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:57:34:321.569 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:57:34:468.287 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:57:34:468.445 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:57:34:468.485 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:57:34:468.509 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:57:34:468.536 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:57:34:468.988 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:57:34:469.033 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:57:34:469.049 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:57:34:469.063 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:57:34:469.078 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:57:38:059.614 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-17 Mon 01:57:38:059.955 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Connect : Start Connect timer 10.0.0.22:8086 2025-02-17 Mon 01:57:38:060.062 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:57:38:060.092 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: SANDESH: Send FAILED: 1739757458059714 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:57:38:060.102 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:57:38:060.111 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:57:38:060.262 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: SANDESH: Send FAILED: 1739757458060045 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 183, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 74, _iter59->first = sent _iter59->second = 328, ] ] ] 2025-02-17 Mon 01:57:38:060.306 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:57:38:060.711 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-17 Mon 01:57:38:060.753 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: -(-1):EvTcpConnectFail 2025-02-17 Mon 01:57:38:060.812 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-17 Mon 01:57:38:060.825 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Connect : EvTcpConnectFail 2025-02-17 Mon 01:57:38:060.943 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-17 Mon 01:57:38:060.977 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Idle 2025-02-17 Mon 01:57:38:061.035 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:38:061.170 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: SANDESH: Send FAILED: 1739757458060901 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 183, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 76, _iter59->first = sent _iter59->second = 328, ] ] ] 2025-02-17 Mon 01:57:38:061.186 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:57:38:061.199 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:38:061.220 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: SANDESH: Send FAILED: 1739757458060923 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:57:38:061.227 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:57:38:061.236 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 01:57:38:061.259 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:38:061.385 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: SANDESH: Send FAILED: 1739757458061026 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 184, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 76, _iter59->first = sent _iter59->second = 328, ] ] ] 2025-02-17 Mon 01:57:38:061.401 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095101318912, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:57:39:317.181 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:57:39:317.344 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:57:39:317.370 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:57:39:317.403 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:57:39:317.432 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:57:39:317.459 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:57:39:317.493 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:57:39:317.508 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:57:39:317.521 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:57:39:317.533 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:57:39:320.474 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:57:39:320.597 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:57:39:320.640 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:57:39:320.933 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:57:39:320.986 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:57:39:321.005 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:57:39:321.526 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:57:39:321.612 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:57:39:321.926 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:57:39:321.956 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:57:39:469.114 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:57:39:469.350 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:57:39:469.388 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:57:39:469.426 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:57:39:469.479 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:57:39:469.812 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:57:39:469.854 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:57:39:469.882 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:57:39:469.901 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:57:39:469.915 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:57:44:320.961 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:57:44:321.092 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:57:44:321.128 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:57:44:321.163 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:57:44:321.208 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:57:44:321.264 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:57:44:321.289 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:57:44:321.320 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:57:44:321.334 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:57:44:321.347 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:57:44:321.358 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:57:44:321.400 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:57:44:321.427 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:57:44:321.643 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:57:44:321.701 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:57:44:321.716 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:57:44:321.967 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:57:44:322.014 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:57:44:322.287 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:57:44:322.333 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:57:44:469.889 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:57:44:470.056 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:57:44:470.092 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:57:44:470.117 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:57:44:470.145 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:57:44:470.501 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:57:44:470.547 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:57:44:470.564 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:57:44:470.579 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:57:44:470.593 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:57:49:321.773 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:57:49:321.937 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:57:49:321.984 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:57:49:322.016 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:57:49:322.046 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:57:49:322.114 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:57:49:322.162 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:57:49:322.193 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:57:49:322.236 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:57:49:322.263 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:57:49:322.277 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:57:49:322.290 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:57:49:322.310 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:57:49:322.328 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:57:49:322.365 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:57:49:323.364 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:57:49:323.482 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:57:49:323.556 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:57:49:323.601 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:57:49:323.869 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:57:49:470.579 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:57:49:470.739 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:57:49:470.788 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:57:49:470.814 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:57:49:470.844 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:57:49:471.162 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:57:49:471.597 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:57:49:471.627 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:57:49:471.642 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:57:49:471.657 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:57:54:322.299 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:57:54:322.442 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:57:54:322.466 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:57:54:322.499 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:57:54:322.530 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:57:54:322.559 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:57:54:322.602 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:57:54:322.619 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:57:54:322.631 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:57:54:322.643 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:57:54:325.260 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:57:54:325.363 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:57:54:325.393 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:57:54:325.419 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:57:54:325.445 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:57:54:326.628 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:57:54:326.666 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:57:54:326.682 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:57:54:326.695 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:57:54:328.265 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:57:54:471.455 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:57:54:471.922 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:57:54:472.020 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:57:54:472.053 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:57:54:472.082 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:57:54:472.120 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:57:54:472.479 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:57:54:472.510 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:57:54:472.526 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:57:54:472.539 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:57:59:322.523 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:57:59:322.691 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:57:59:322.718 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:57:59:322.775 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:57:59:322.814 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:57:59:322.843 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:57:59:322.889 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:57:59:322.907 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:57:59:322.949 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:57:59:322.961 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:57:59:326.806 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:57:59:326.925 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:57:59:326.955 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:57:59:326.980 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:57:59:327.349 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:57:59:327.401 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:57:59:327.416 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:57:59:327.430 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:57:59:337.039 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:57:59:337.619 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:57:59:472.201 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:57:59:472.529 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:57:59:472.598 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:57:59:472.631 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:57:59:472.657 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:57:59:472.812 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:57:59:472.909 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:57:59:472.930 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:57:59:472.982 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:57:59:472.998 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:58:03:937.567 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:58:03:937.676 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: SANDESH: Send FAILED: 1739757483937427 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 407 dequeues = 407 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 445 dequeues = 445 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739757458060976 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739757458061257 ] max_sm_queue_count = 5 ] 2025-02-17 Mon 01:58:03:937.702 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:58:03:937.712 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:58:03:937.869 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: SANDESH: Send FAILED: 1739757483937539 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 184, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 79, _iter59->first = sent _iter59->second = 328, ] ] ] 2025-02-17 Mon 01:58:03:937.888 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095076136704, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:58:04:322.781 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:58:04:322.979 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d30840 2025-02-17 Mon 01:58:04:323.018 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:58:04:323.073 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:58:04:323.133 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d300d0 2025-02-17 Mon 01:58:04:323.152 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d29070 2025-02-17 Mon 01:58:04:323.163 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:58:04:323.194 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:58:04:323.228 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2fcd0 2025-02-17 Mon 01:58:04:323.243 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d2f620 2025-02-17 Mon 01:58:04:327.440 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:58:04:327.546 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:58:04:327.582 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:58:04:327.607 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:58:04:327.975 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31ac0 2025-02-17 Mon 01:58:04:328.009 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d31180 2025-02-17 Mon 01:58:04:328.026 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33680 2025-02-17 Mon 01:58:04:328.040 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32400 2025-02-17 Mon 01:58:04:337.699 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:58:04:338.195 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d32d40 2025-02-17 Mon 01:58:04:472.972 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:58:04:473.110 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:58:04:473.151 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:58:04:473.193 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:58:04:473.236 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:58:04:474.940 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35b80 2025-02-17 Mon 01:58:04:474.985 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d34900 2025-02-17 Mon 01:58:04:475.001 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d33fc0 2025-02-17 Mon 01:58:04:475.015 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d35240 2025-02-17 Mon 01:58:04:475.029 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: -1 0x1d364c0 2025-02-17 Mon 01:58:08:064.628 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095194330880, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-17 Mon 01:58:08:064.978 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095194330880, Pid 1]: Connect : Start Connect timer 10.0.0.56:8086 2025-02-17 Mon 01:58:08:065.111 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095194330880, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:58:08:065.192 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095194330880, Pid 1]: SANDESH: Send FAILED: 1739757488064714 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:58:08:065.208 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095194330880, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:58:08:065.231 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095194330880, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:58:08:065.491 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095194330880, Pid 1]: SANDESH: Send FAILED: 1739757488065091 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 47 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 184, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 81, _iter59->first = sent _iter59->second = 328, ] ] ] 2025-02-17 Mon 01:58:08:065.526 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095194330880, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:58:08:068.478 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-17 Mon 01:58:08:068.499 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: -(-1):EvTcpConnectFail 2025-02-17 Mon 01:58:08:075.111 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-17 Mon 01:58:08:075.171 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Connect : EvTcpConnectFail 2025-02-17 Mon 01:58:08:075.421 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-17 Mon 01:58:08:075.494 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Idle 2025-02-17 Mon 01:58:08:075.623 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:58:08:075.809 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: SANDESH: Send FAILED: 1739757488075320 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 184, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 83, _iter59->first = sent _iter59->second = 328, ] ] ] 2025-02-17 Mon 01:58:08:075.837 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:58:08:075.862 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:58:08:075.889 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: SANDESH: Send FAILED: 1739757488075378 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:58:08:075.899 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:58:08:075.912 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 01:58:08:075.942 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:58:08:076.202 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: SANDESH: Send FAILED: 1739757488075591 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 169 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 185, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 83, _iter59->first = sent _iter59->second = 328, ] ] ] 2025-02-17 Mon 01:58:08:076.225 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:58:08:938.845 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095194330880, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:58:08:938.933 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095194330880, Pid 1]: SANDESH: Send FAILED: 1739757488938808 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-02-17 Mon 01:58:08:938.944 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095194330880, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-02-17 Mon 01:58:09:323.059 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d30840 2025-02-17 Mon 01:58:09:323.245 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d30840 2025-02-17 Mon 01:58:09:323.260 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:323.285 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:323.296 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d300d0 2025-02-17 Mon 01:58:09:323.341 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d29070 2025-02-17 Mon 01:58:09:323.414 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d300d0 2025-02-17 Mon 01:58:09:323.423 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:323.433 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:323.452 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d29070 2025-02-17 Mon 01:58:09:323.458 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:323.466 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:323.473 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2fcd0 2025-02-17 Mon 01:58:09:323.520 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d2f620 2025-02-17 Mon 01:58:09:323.585 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:323.597 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:323.627 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d2fcd0 2025-02-17 Mon 01:58:09:323.634 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:323.643 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:323.651 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d2f620 2025-02-17 Mon 01:58:09:323.656 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:323.665 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:323.690 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:323.710 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:323.720 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:323.732 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:323.741 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:331.030 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:58:09:331.102 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: SANDESH: Send FAILED: 1739757489323808 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.53:6379::Down.10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 01:58:09:331.126 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:58:09:331.307 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:331.339 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:331.381 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:331.391 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:331.406 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:331.415 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:331.424 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31ac0 2025-02-17 Mon 01:58:09:331.517 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d31180 2025-02-17 Mon 01:58:09:331.549 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33680 2025-02-17 Mon 01:58:09:331.577 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32400 2025-02-17 Mon 01:58:09:331.643 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:331.665 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:332.125 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d31ac0 2025-02-17 Mon 01:58:09:332.147 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:332.166 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:332.177 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d31180 2025-02-17 Mon 01:58:09:332.183 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:332.202 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:332.211 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d33680 2025-02-17 Mon 01:58:09:332.216 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:332.224 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:332.233 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d32400 2025-02-17 Mon 01:58:09:332.239 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:332.246 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:332.756 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:332.793 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:332.810 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:332.820 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:332.843 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:336.381 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095194330880, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:58:09:336.440 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095194330880, Pid 1]: SANDESH: Send FAILED: 1739757489332892 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 01:58:09:336.451 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095194330880, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:58:09:336.578 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:336.613 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:336.636 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:336.652 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:336.661 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:336.681 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:336.696 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:338.265 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d32d40 2025-02-17 Mon 01:58:09:338.746 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d32d40 2025-02-17 Mon 01:58:09:338.756 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:338.790 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:339.129 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:339.149 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:339.171 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:475.031 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35b80 2025-02-17 Mon 01:58:09:475.179 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d34900 2025-02-17 Mon 01:58:09:475.223 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d33fc0 2025-02-17 Mon 01:58:09:475.253 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d35240 2025-02-17 Mon 01:58:09:475.280 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1d364c0 2025-02-17 Mon 01:58:09:475.489 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d35b80 2025-02-17 Mon 01:58:09:475.498 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:475.524 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:475.560 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d34900 2025-02-17 Mon 01:58:09:475.566 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:475.575 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:475.611 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d35240 2025-02-17 Mon 01:58:09:475.617 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:475.626 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:475.687 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d33fc0 2025-02-17 Mon 01:58:09:475.694 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:475.703 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:475.712 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: RAC_Connect status: 0 0x1d364c0 2025-02-17 Mon 01:58:09:475.717 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:475.725 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:475.895 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:476.015 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:476.043 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:476.059 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:476.069 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:477.019 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:58:09:477.085 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: SANDESH: Send FAILED: 1739757489476097 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 01:58:09:477.108 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095118108416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:58:09:477.134 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:477.164 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:477.175 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:477.194 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:477.208 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:477.217 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:477.230 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:477.239 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:477.254 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:477.266 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:33:939.695 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 65 without seqno , total = 0 2025-02-17 Mon 01:58:33:939.781 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:58:33:939.851 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: SANDESH: Send FAILED: 1739757513939561 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 418 dequeues = 418 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 459 dequeues = 459 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739757488075491 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739757489477008 ] max_sm_queue_count = 5 ] 2025-02-17 Mon 01:58:33:939.875 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:58:33:939.888 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:58:33:940.096 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: SANDESH: Send FAILED: 1739757513939677 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 229 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 50 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 245, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 90, _iter59->first = sent _iter59->second = 328, ] ] ] 2025-02-17 Mon 01:58:33:940.123 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095202727680, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:58:38:078.840 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095113910016, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-17 Mon 01:58:38:079.048 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095113910016, Pid 1]: Connect : Start Connect timer 10.0.0.53:8086 2025-02-17 Mon 01:58:38:079.165 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095113910016, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:58:38:079.202 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095113910016, Pid 1]: SANDESH: Send FAILED: 1739757518078917 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.53:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 01:58:38:079.211 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095113910016, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:58:38:079.220 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095113910016, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:58:38:083.754 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095113910016, Pid 1]: SANDESH: Send FAILED: 1739757518079150 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 229 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 52 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 245, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 92, _iter59->first = sent _iter59->second = 328, ] ] ] 2025-02-17 Mon 01:58:38:083.833 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095113910016, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:58:38:086.341 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.56:44102::10.0.0.53:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-17 Mon 01:58:38:086.365 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:44102::10.0.0.53:8086(-1) OnSessionEvent TCP Connected 2025-02-17 Mon 01:58:38:086.374 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095346399360, Pid 1]: 10.0.0.56:44102::10.0.0.53:8086(-1):EvTcpConnected 2025-02-17 Mon 01:58:38:095.250 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-17 Mon 01:58:38:095.297 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-17 Mon 01:58:38:095.354 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: ClientInit 2025-02-17 Mon 01:58:38:095.367 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2738-1.:contrail-query-engine:0:Database count 7 2025-02-17 Mon 01:58:38:095.462 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:58:38:095.472 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 01:58:38:095.498 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: SANDESH: Send FAILED: 1739757518095343 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.53:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 01:58:38:095.506 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 01:58:38:095.516 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:58:38:095.523 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-17 Mon 01:58:38:095.538 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:58:38:095.545 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-17 Mon 01:58:38:095.687 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: SANDESH: Send FAILED: 1739757518095451 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 7 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 229 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 53 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 246, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 94, _iter59->first = sent _iter59->second = 328, ] ] ] 2025-02-17 Mon 01:58:38:095.711 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095067739904, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-17 Mon 01:58:38:104.274 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: OnMessage control in state: ClientInit 2025-02-17 Mon 01:58:38:104.334 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-17 Mon 01:58:38:104.344 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-17 Mon 01:58:38:104.372 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Received Ctrl Message with size 0 2025-02-17 Mon 01:58:38:104.593 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 7 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3043869 calls = 327 average_bytes = 9308 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 418 dequeues = 418 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 459 dequeues = 459 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739757488075491 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739757489477008 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 1 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 229 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 250 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 53 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 246, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 94, _iter59->first = sent _iter59->second = 328, ] ] max_sm_queue_count = 5 proxy seq 304 2025-02-17 Mon 01:58:38:104.630 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-17 Mon 01:58:38:111.167 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.53:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 48 2025-02-17 Mon 01:58:38:111.224 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-17 Mon 01:58:38:111.273 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2738-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 65 2025-02-17 Mon 01:58:38:111.285 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-02-17 Mon 01:58:38:111.292 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-17 Mon 01:58:38:111.330 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Established 2025-02-17 Mon 01:58:43:804.649 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: OnMessage control in state: Established 2025-02-17 Mon 01:58:43:804.747 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2025-02-17 Mon 01:58:43:804.768 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Established : EvSandeshMessageRecv 2025-02-17 Mon 01:58:43:804.812 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: Received Ctrl Message with size 3 2025-02-17 Mon 01:58:43:804.826 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 305, total = 0 2025-02-17 Mon 01:58:43:804.838 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 50, total = 0 2025-02-17 Mon 01:58:43:804.861 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 65, total = 0 2025-02-17 Mon 01:58:43:804.870 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-17 Mon 01:59:33:945.814 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 66 without seqno , total = 0 2025-02-17 Mon 02:00:33:949.845 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095071938304, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 67 without seqno , total = 0 2025-02-17 Mon 02:01:33:951.973 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095113910016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 68 without seqno , total = 0 2025-02-17 Mon 02:02:26:282.680 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Processing scm::EvStop in state Established 2025-02-17 Mon 02:02:26:282.805 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Idle 2025-02-17 Mon 02:02:26:282.912 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 02:02:26:282.924 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 02:02:26:283.076 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: SANDESH: Send FAILED: 1739757746282900 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 7 pid = 1 http_port = 8091 start_time = 1739753573299366 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 4896 calls = 8 average_bytes = 612 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3269053 calls = 349 average_bytes = 9366 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 68 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 229 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 266 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 54 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 246, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 96, _iter59->first = sent _iter59->second = 353, ] ] ] 2025-02-17 Mon 02:02:26:283.106 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 140095063541504, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 02:02:56:930.029 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: http-server-port 8091 2025-02-17 Mon 02:02:56:930.121 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Max-tasks 16 2025-02-17 Mon 02:02:56:930.129 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Max-slice 100 2025-02-17 Mon 02:02:56:930.136 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Collectors 10.0.0.56:8086 2025-02-17 Mon 02:02:56:930.141 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Collectors 10.0.0.53:8086 2025-02-17 Mon 02:02:56:930.146 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Collectors 10.0.0.22:8086 2025-02-17 Mon 02:02:56:930.172 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: SANDESH: ROLE : Generator 2025-02-17 Mon 02:02:56:930.178 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-17 Mon 02:02:56:930.184 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2738-1. 2025-02-17 Mon 02:02:56:930.189 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-17 Mon 02:02:56:930.194 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-17 Mon 02:02:56:930.204 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-17 Mon 02:02:56:930.210 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-17 Mon 02:02:56:930.892 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: HTTP Introspect Init 2025-02-17 Mon 02:02:56:931.028 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-17 Mon 02:02:56:931.099 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Sandesh Http Server Port: 8091 2025-02-17 Mon 02:02:56:982.002 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-17 Mon 02:02:56:982.060 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-17 Mon 02:02:56:982.202 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Idle 2025-02-17 Mon 02:02:56:982.407 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: SANDESH: No Client: 1739757776982319 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739757776982216 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 02:02:56:982.604 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-17 Mon 02:02:56:982.623 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-17 Mon 02:02:56:982.640 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Cassandra Servers: 10.0.0.56:9042 10.0.0.53:9042 10.0.0.22:9042 2025-02-17 Mon 02:02:56:982.773 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: Processing scm::EvStart in state Idle 2025-02-17 Mon 02:02:56:982.883 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: Disconnect 2025-02-17 Mon 02:02:56:982.970 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-17 Mon 02:02:56:983.310 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: Connect : Start Connect timer 10.0.0.56:8086 2025-02-17 Mon 02:02:56:983.432 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 02:02:56:983.463 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: SANDESH: Send FAILED: 1739757776982872 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-17 Mon 02:02:56:983.476 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 02:02:56:983.490 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 02:02:56:983.876 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: SANDESH: Send FAILED: 1739757776982957 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739757776982216 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 02:02:56:983.907 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 02:02:56:983.920 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 02:02:56:984.023 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: SANDESH: Send FAILED: 1739757776983021 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739757776982216 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 02:02:56:984.038 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 02:02:56:984.057 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 02:02:56:984.080 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: SANDESH: Send FAILED: 1739757776983045 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-17 Mon 02:02:56:984.091 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 02:02:56:984.102 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 02:02:56:984.189 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: SANDESH: Send FAILED: 1739757776983416 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739757776982216 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 02:02:56:984.208 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 02:02:57:027.846 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-17 Mon 02:02:57:032.965 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 02:02:57:033.035 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: SANDESH: Send FAILED: 1739757777031557 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-17 Mon 02:02:57:033.047 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 02:02:57:036.262 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.56:46980::10.0.0.56:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2025-02-17 Mon 02:02:57:036.296 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: 10.0.0.56:46980::10.0.0.56:8086(-1) OnSessionEvent TCP Connected 2025-02-17 Mon 02:02:57:036.305 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: 10.0.0.56:46980::10.0.0.56:8086(-1):EvTcpConnected 2025-02-17 Mon 02:02:57:036.340 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: RAC_Connect status: 0 0x1321280 2025-02-17 Mon 02:02:57:036.346 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:02:57:036.358 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:02:57:036.372 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: RAC_Connect status: 0 0x1321980 2025-02-17 Mon 02:02:57:036.378 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:02:57:036.386 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:02:57:036.395 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: RAC_Connect status: 0 0x1322190 2025-02-17 Mon 02:02:57:036.401 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:02:57:036.408 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:02:57:036.418 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: RAC_Connect status: 0 0x1322a10 2025-02-17 Mon 02:02:57:036.423 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:02:57:036.438 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:02:57:036.448 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: RAC_Connect status: 0 0x1323290 2025-02-17 Mon 02:02:57:036.453 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:02:57:036.461 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:02:57:036.471 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: RAC_Connect status: 0 0x135eca0 2025-02-17 Mon 02:02:57:036.476 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:02:57:036.484 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:02:57:036.494 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: RAC_Connect status: 0 0x135f520 2025-02-17 Mon 02:02:57:036.499 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:02:57:036.506 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:02:57:036.516 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: RAC_Connect status: 0 0x1360620 2025-02-17 Mon 02:02:57:036.522 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:02:57:036.535 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:02:57:036.544 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: RAC_Connect status: 0 0x135fda0 2025-02-17 Mon 02:02:57:036.549 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:02:57:036.557 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:02:57:036.566 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: RAC_Connect status: 0 0x1360ea0 2025-02-17 Mon 02:02:57:036.571 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:02:57:036.579 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:02:57:036.588 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: RAC_Connect status: 0 0x1361720 2025-02-17 Mon 02:02:57:036.593 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:02:57:036.601 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:02:57:036.611 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: RAC_Connect status: 0 0x13630a0 2025-02-17 Mon 02:02:57:036.616 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:02:57:036.627 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:02:57:036.636 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: RAC_Connect status: 0 0x1363920 2025-02-17 Mon 02:02:57:036.641 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:02:57:036.649 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:02:57:036.667 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: RAC_Connect status: 0 0x1362820 2025-02-17 Mon 02:02:57:036.675 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:02:57:036.683 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:02:57:036.692 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: RAC_Connect status: 0 0x1361fa0 2025-02-17 Mon 02:02:57:036.698 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:02:57:036.705 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:02:57:036.894 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-17 Mon 02:02:57:036.921 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-17 Mon 02:02:57:036.964 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: ClientInit 2025-02-17 Mon 02:02:57:036.985 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2738-1.:contrail-query-engine:0:Database count 1 2025-02-17 Mon 02:02:57:037.087 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 02:02:57:037.097 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 02:02:57:037.118 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: SANDESH: Send FAILED: 1739757777036956 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-17 Mon 02:02:57:037.125 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 02:02:57:037.134 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 02:02:57:037.141 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-17 Mon 02:02:57:037.174 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 02:02:57:037.182 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-17 Mon 02:02:57:037.263 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:02:57:037.286 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:02:57:037.302 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:02:57:037.315 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:02:57:037.327 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:02:57:037.336 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:02:57:037.348 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:02:57:037.357 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:02:57:037.369 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:02:57:037.378 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:02:57:037.419 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:02:57:037.448 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:02:57:037.465 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:02:57:037.481 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:02:57:037.496 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:02:57:037.302 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: SANDESH: Send FAILED: 1739757777037078 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739757776982216 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 02:02:57:037.575 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-17 Mon 02:02:57:037.589 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 02:02:57:037.609 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 02:02:57:037.649 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: SANDESH: Send FAILED: 1739757777037409 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 02:02:57:037.667 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 02:02:57:037.677 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 02:02:57:037.684 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 02:02:57:037.720 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: SANDESH: Send FAILED: 1739757777037439 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.53:6379::Down.10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 02:02:57:037.728 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 02:02:57:040.921 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:02:57:040.951 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:02:57:040.977 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:02:57:040.986 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:02:57:041.000 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:02:57:041.009 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:02:57:041.022 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:02:57:041.031 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:02:57:041.043 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:02:57:041.053 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:02:57:041.066 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:02:57:041.075 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:02:57:041.088 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:02:57:041.109 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:02:57:041.563 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:02:57:041.583 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:02:57:041.600 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:02:57:041.609 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:02:57:041.622 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:02:57:041.638 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:02:57:041.689 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:02:57:041.735 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:02:57:041.751 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:02:57:041.766 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:02:57:041.780 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:02:57:041.803 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:02:57:041.825 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:02:57:041.838 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:02:57:041.850 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:02:57:041.863 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:02:57:041.928 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 02:02:57:041.944 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 02:02:57:041.966 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: SANDESH: Send FAILED: 1739757777041716 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 02:02:57:041.975 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 02:02:57:041.992 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 02:02:57:042.005 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 02:02:57:042.026 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: SANDESH: Send FAILED: 1739757777041819 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 02:02:57:042.033 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 02:02:57:042.166 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: OnMessage control in state: ClientInit 2025-02-17 Mon 02:02:57:042.187 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-17 Mon 02:02:57:042.196 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-17 Mon 02:02:57:042.219 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: Received Ctrl Message with size 0 2025-02-17 Mon 02:02:57:042.349 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2738-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739757776982216 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2025-02-17 Mon 02:02:57:042.371 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-17 Mon 02:02:57:042.393 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2738-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2025-02-17 Mon 02:02:57:042.404 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-17 Mon 02:02:57:042.412 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-17 Mon 02:02:57:042.420 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-17 Mon 02:02:57:042.447 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: Established 2025-02-17 Mon 02:03:56:983.070 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791706339072, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-17 Mon 02:04:56:984.062 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-17 Mon 02:05:56:985.234 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139790645192448, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-17 Mon 02:06:56:985.936 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791718934272, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-17 Mon 02:07:56:986.556 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791718934272, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-17 Mon 02:08:56:987.150 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791718934272, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-17 Mon 02:09:56:987.897 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-17 Mon 02:10:56:988.537 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791794714368, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-17 Mon 02:11:56:989.193 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-17 Mon 02:12:56:989.903 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791706339072, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-02-17 Mon 02:13:56:990.792 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-02-17 Mon 02:14:56:991.648 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-02-17 Mon 02:15:56:992.203 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791803111168, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-02-17 Mon 02:16:56:993.046 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139790649390848, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-02-17 Mon 02:17:56:993.587 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139790640994048, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-02-17 Mon 02:18:56:994.156 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-02-17 Mon 02:19:56:994.869 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139790649390848, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-02-17 Mon 02:20:42:146.622 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.56:8091::10.0.0.56:44922 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-17 Mon 02:20:44:912.804 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.56:8091::10.0.0.56:46550 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-17 Mon 02:20:56:995.474 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-02-17 Mon 02:21:56:996.184 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791794714368, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-02-17 Mon 02:22:14:696.898 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.56:8091::10.0.0.56:37588 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-17 Mon 02:22:37:773.988 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.56 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 773 2025-02-17 Mon 02:22:37:774.609 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID 0e3af398-ecd6-11ef-9110-00000a000038 src/contrail-analytics/contrail-query-engine/query.cc 1166 2025-02-17 Mon 02:22:37:774.681 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 :0:json query is: { end_time: 1739758957682771 , enqueue_time: 1739758957772772 , query_metadata: {"enqueue_time": 1739758957772285} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1739758939344830 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-17 Mon 02:22:37:774.925 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 880 2025-02-17 Mon 02:22:37:774.947 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 931 2025-02-17 Mon 02:22:37:775.043 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791710537472, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 :0:json query is: { end_time: 1739758957682771 , enqueue_time: 1739758957772772 , query_metadata: {"enqueue_time": 1739758957772285} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1739758939344830 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-17 Mon 02:22:37:776.133 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for 0e3af398-ecd6-11ef-9110-00000a000038 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 940 2025-02-17 Mon 02:22:38:013.180 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:013.247 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:013.261 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:013.272 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:013.285 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 UNION between tables of sizes 0 and 34 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:013.345 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 Resulting size of set 34 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:013.359 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 UNION between tables of sizes 34 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:013.383 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 Resulting size of set 34 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:013.395 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 UNION between tables of sizes 34 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:013.411 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 Resulting size of set 34 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:013.475 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 :0:json query is: { end_time: 1739758957682771 , enqueue_time: 1739758957772772 , query_metadata: {"enqueue_time": 1739758957772285} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1739758939344830 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-17 Mon 02:22:38:013.724 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:013.870 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:014.029 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:014.233 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:014.453 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:014.664 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:014.834 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:014.975 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:015.181 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:015.391 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:015.582 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:015.718 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:015.849 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:015.981 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:016.532 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:016.750 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:016.983 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:017.155 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:017.341 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:017.479 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:017.620 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:017.780 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:017.921 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:018.117 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:018.324 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:018.546 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:018.766 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:018.963 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:019.117 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:019.258 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:019.421 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:019.578 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:019.742 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:019.876 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:031.261 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139790649390848, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:031.302 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139790649390848, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:031.312 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139790649390848, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:031.320 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139790649390848, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:031.328 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139790649390848, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:031.335 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139790649390848, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:031.343 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139790649390848, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:031.350 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139790649390848, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:031.357 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139790649390848, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:031.365 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139790649390848, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:044.121 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:044.157 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:044.177 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:044.185 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:044.194 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 UNION between tables of sizes 0 and 10 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:044.213 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 Resulting size of set 10 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:044.223 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 UNION between tables of sizes 10 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:044.236 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 Resulting size of set 10 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:044.245 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 UNION between tables of sizes 10 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:044.256 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QELog: 0e3af398-ecd6-11ef-9110-00000a000038 Resulting size of set 10 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:044.523 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:044.670 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:044.802 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:045.071 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:045.208 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:045.344 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:045.482 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:045.625 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:045.757 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:045.889 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QEQueryLog: 0e3af398-ecd6-11ef-9110-00000a000038 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:046.153 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 44 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 520 2025-02-17 Mon 02:22:38:047.322 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 44 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 528 2025-02-17 Mon 02:22:38:048.426 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":6, "count":44} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 563 2025-02-17 Mon 02:22:38:048.568 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: [SYS_INFO]: QELog: Finished: QID 0e3af398-ecd6-11ef-9110-00000a000038 Table ObjectVNTable Time(ms) 273 RedisTime(ms) 1 MergeTime(ms) 0 Rows 44 EnQ-delay1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 644 2025-02-17 Mon 02:22:38:048.681 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 676 2025-02-17 Mon 02:22:38:048.812 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791702140672, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-2738-1. table = ObjectVNTable query_stats= [ qid = 0e3af398-ecd6-11ef-9110-00000a000038 where = [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 219,0, 268,0, 254,0, chunk_select_time = 0,6, 0,1, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 273 rows = 44 enq_delay = 1 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 636 2025-02-17 Mon 02:22:40:931.778 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.56 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 773 2025-02-17 Mon 02:22:40:932.052 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID 101d700a-ecd6-11ef-9fed-00000a000038 src/contrail-analytics/contrail-query-engine/query.cc 1166 2025-02-17 Mon 02:22:40:932.089 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QEQueryLog: 101d700a-ecd6-11ef-9fed-00000a000038 :0:json query is: { end_time: 1739758960918550 , enqueue_time: 1739758960931435 , query_metadata: {"enqueue_time": 1739758960931331} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1739758939344830 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "7a9b2a95-d775-4174-901b-a3b61556288c", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-17 Mon 02:22:40:932.259 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 880 2025-02-17 Mon 02:22:40:932.275 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 931 2025-02-17 Mon 02:22:40:932.328 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for 101d700a-ecd6-11ef-9fed-00000a000038 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 940 2025-02-17 Mon 02:22:40:932.422 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: [SYS_INFO]: QEQueryLog: 101d700a-ecd6-11ef-9fed-00000a000038 :0:json query is: { end_time: 1739758960918550 , enqueue_time: 1739758960931435 , query_metadata: {"enqueue_time": 1739758960931331} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1739758939344830 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "7a9b2a95-d775-4174-901b-a3b61556288c", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-17 Mon 02:22:41:001.833 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791663769344, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:41:001.891 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791663769344, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:41:001.915 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791663769344, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 UNION between tables of sizes 0 and 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:41:001.928 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791663769344, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:41:001.939 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791663769344, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 UNION between tables of sizes 5 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:41:001.950 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791663769344, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:41:001.959 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791663769344, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 UNION between tables of sizes 5 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:41:001.969 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791663769344, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:41:001.978 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791663769344, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 UNION between tables of sizes 5 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:41:001.988 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791663769344, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:41:002.271 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791663769344, Pid 1]: [SYS_INFO]: QEQueryLog: 101d700a-ecd6-11ef-9fed-00000a000038 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 7a9b2a95-d775-4174-901b-a3b61556288c src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:41:002.416 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791663769344, Pid 1]: [SYS_INFO]: QEQueryLog: 101d700a-ecd6-11ef-9fed-00000a000038 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 7a9b2a95-d775-4174-901b-a3b61556288c src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:41:002.564 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791663769344, Pid 1]: [SYS_INFO]: QEQueryLog: 101d700a-ecd6-11ef-9fed-00000a000038 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 7a9b2a95-d775-4174-901b-a3b61556288c src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:41:002.697 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791663769344, Pid 1]: [SYS_INFO]: QEQueryLog: 101d700a-ecd6-11ef-9fed-00000a000038 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 7a9b2a95-d775-4174-901b-a3b61556288c src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:41:002.838 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791663769344, Pid 1]: [SYS_INFO]: QEQueryLog: 101d700a-ecd6-11ef-9fed-00000a000038 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 7a9b2a95-d775-4174-901b-a3b61556288c src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:41:006.093 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:41:006.178 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:41:006.194 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:41:006.207 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:41:006.220 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:41:006.233 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:41:006.246 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:41:006.257 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:41:006.269 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:41:006.281 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:41:006.359 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QEQueryLog: 101d700a-ecd6-11ef-9fed-00000a000038 :0:json query is: { end_time: 1739758960918550 , enqueue_time: 1739758960931435 , query_metadata: {"enqueue_time": 1739758960931331} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1739758939344830 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "7a9b2a95-d775-4174-901b-a3b61556288c", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-17 Mon 02:22:41:006.614 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 UNION between tables of sizes 0 and 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:41:006.640 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:41:006.653 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:41:006.666 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:41:006.677 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:41:006.689 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:41:006.700 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:41:006.712 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:41:006.721 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:41:006.729 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: 101d700a-ecd6-11ef-9fed-00000a000038 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:41:006.934 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QEQueryLog: 101d700a-ecd6-11ef-9fed-00000a000038 ObjectVMTable :2:Column ObjectTypeName1 Table ObjectVMTable object_id 7a9b2a95-d775-4174-901b-a3b61556288c src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:41:007.092 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 6 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 520 2025-02-17 Mon 02:22:41:007.182 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 6 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 528 2025-02-17 Mon 02:22:41:007.331 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791723132672, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":6} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 563 2025-02-17 Mon 02:22:41:007.560 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791803111168, Pid 1]: [SYS_INFO]: QELog: Finished: QID 101d700a-ecd6-11ef-9fed-00000a000038 Table ObjectVMTable Time(ms) 75 RedisTime(ms) 0 MergeTime(ms) 0 Rows 6 EnQ-delay0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 644 2025-02-17 Mon 02:22:41:007.618 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791803111168, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 676 2025-02-17 Mon 02:22:41:007.689 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791803111168, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-2738-1. table = ObjectVMTable query_stats= [ qid = 101d700a-ecd6-11ef-9fed-00000a000038 where = [[{"name": "ObjectId", "value": "7a9b2a95-d775-4174-901b-a3b61556288c", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 73,0, 69,0, 73,0, chunk_select_time = 0,0, 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 75 rows = 6 enq_delay = 0 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 636 2025-02-17 Mon 02:22:56:998.028 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-02-17 Mon 02:23:56:998.838 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139790645192448, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-02-17 Mon 02:24:56:999.525 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791718934272, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-02-17 Mon 02:25:57:000.155 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-02-17 Mon 02:26:57:000.782 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791697942272, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-02-17 Mon 02:27:57:001.492 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791714735872, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-02-17 Mon 02:28:57:002.152 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791693743872, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2025-02-17 Mon 02:29:39:762.725 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.56:8091::10.0.0.56:56026 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-17 Mon 02:29:57:002.810 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791807309568, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2025-02-17 Mon 02:30:57:003.868 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139790649390848, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2025-02-17 Mon 02:31:55:061.486 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791955171456, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.56:8091::10.0.0.56:49304 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-17 Mon 02:31:57:004.610 UTC cn-jenkins-deploy-platform-ansible-os-2738-1 [Thread 139791794714368, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0