2025-02-17 Mon 00:52:53:136.466 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: http-server-port 8091 2025-02-17 Mon 00:52:53:136.573 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Max-tasks 16 2025-02-17 Mon 00:52:53:136.582 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Max-slice 100 2025-02-17 Mon 00:52:53:136.588 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Collectors 10.0.0.56:8086 2025-02-17 Mon 00:52:53:136.593 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Collectors 10.0.0.53:8086 2025-02-17 Mon 00:52:53:136.597 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Collectors 10.0.0.22:8086 2025-02-17 Mon 00:52:53:136.617 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: SANDESH: ROLE : Generator 2025-02-17 Mon 00:52:53:136.623 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-17 Mon 00:52:53:136.628 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2738-2. 2025-02-17 Mon 00:52:53:136.633 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-17 Mon 00:52:53:136.638 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-17 Mon 00:52:53:136.643 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-17 Mon 00:52:53:136.649 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-17 Mon 00:52:53:137.230 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: HTTP Introspect Init 2025-02-17 Mon 00:52:53:137.344 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-17 Mon 00:52:53:137.400 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Sandesh Http Server Port: 8091 2025-02-17 Mon 00:52:53:137.439 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-17 Mon 00:52:53:137.447 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-17 Mon 00:52:53:137.562 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Idle 2025-02-17 Mon 00:52:53:137.756 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: SANDESH: No Client: 1739753573137660 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739753573137576 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 00:52:53:137.884 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-17 Mon 00:52:53:137.898 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-17 Mon 00:52:53:137.910 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Cassandra Servers: 10.0.0.56:9042 10.0.0.53:9042 10.0.0.22:9042 2025-02-17 Mon 00:52:53:141.112 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: Processing scm::EvStart in state Idle 2025-02-17 Mon 00:52:53:141.178 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: Disconnect 2025-02-17 Mon 00:52:53:141.249 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-17 Mon 00:52:53:141.777 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: Connect : Start Connect timer 10.0.0.22:8086 2025-02-17 Mon 00:52:53:141.858 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 00:52:53:141.880 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: SANDESH: Send FAILED: 1739753573141170 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-17 Mon 00:52:53:141.889 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 00:52:53:141.897 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 00:52:53:142.004 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: SANDESH: Send FAILED: 1739753573141239 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739753573137576 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 00:52:53:142.027 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 00:52:53:142.039 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 00:52:53:142.110 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: SANDESH: Send FAILED: 1739753573141308 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739753573137576 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 00:52:53:142.123 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 00:52:53:142.132 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 00:52:53:142.147 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: SANDESH: Send FAILED: 1739753573141329 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-17 Mon 00:52:53:142.153 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 00:52:53:142.161 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 00:52:53:142.223 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: SANDESH: Send FAILED: 1739753573141845 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739753573137576 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 00:52:53:142.235 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 00:52:53:159.394 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-17 Mon 00:52:53:159.430 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:52:53:159.494 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:52:53:159.564 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243188102912, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 00:52:53:159.595 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243188102912, Pid 1]: SANDESH: Send FAILED: 1739753573159467 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-17 Mon 00:52:53:159.604 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243188102912, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 00:52:58:162.199 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-17 Mon 00:52:58:162.261 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:52:58:162.307 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:03:164.032 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-17 Mon 00:53:03:164.082 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:03:164.105 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:08:166.267 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-02-17 Mon 00:53:08:166.338 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:08:166.362 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:13:332.584 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-17 Mon 00:53:13:332.622 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:13:332.645 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:18:519.445 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-17 Mon 00:53:18:519.568 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:18:519.592 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:23:541.240 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-17 Mon 00:53:23:541.290 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:23:541.320 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:28:564.247 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-17 Mon 00:53:28:564.361 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:28:564.394 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 7 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:33:579.818 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-17 Mon 00:53:33:579.880 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:33:579.901 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 8 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:38:611.570 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-17 Mon 00:53:38:611.645 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:38:611.667 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 9 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:43:681.493 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-17 Mon 00:53:43:681.539 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:43:681.560 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 10 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:48:722.807 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-02-17 Mon 00:53:48:722.850 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:48:722.872 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 11 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:53:748.416 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-02-17 Mon 00:53:53:748.498 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 12 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-02-17 Mon 00:53:58:775.075 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-17 Mon 00:54:03:782.983 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243175507712, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 00:54:03:783.070 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243175507712, Pid 1]: SANDESH: Send FAILED: 1739753643782888 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-17 Mon 00:54:03:783.081 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243175507712, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 00:54:03:783.019 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-17 Mon 00:54:03:783.196 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: -(-1):EvTcpConnectFail 2025-02-17 Mon 00:54:03:783.242 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: RAC_Connect status: 0 0xd451a0 2025-02-17 Mon 00:54:03:783.249 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:783.250 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-17 Mon 00:54:03:783.298 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: Connect : EvTcpConnectFail 2025-02-17 Mon 00:54:03:783.270 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:783.408 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: RAC_Connect status: 0 0xd45ae0 2025-02-17 Mon 00:54:03:783.415 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:783.425 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:783.434 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: RAC_Connect status: 0 0xd46420 2025-02-17 Mon 00:54:03:783.440 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:783.444 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-17 Mon 00:54:03:783.449 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:783.499 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: Idle 2025-02-17 Mon 00:54:03:783.503 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: RAC_Connect status: 0 0xd46d60 2025-02-17 Mon 00:54:03:783.520 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:783.551 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:783.567 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: RAC_Connect status: 0 0xd476a0 2025-02-17 Mon 00:54:03:783.582 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:783.571 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 00:54:03:783.595 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:783.626 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: RAC_Connect status: 0 0xd435f0 2025-02-17 Mon 00:54:03:783.633 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:783.649 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:783.662 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: RAC_Connect status: 0 0xd47fe0 2025-02-17 Mon 00:54:03:783.670 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:783.684 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:783.697 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: RAC_Connect status: 0 0xd48920 2025-02-17 Mon 00:54:03:783.704 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:783.718 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:783.721 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: SANDESH: Send FAILED: 1739753643783390 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739753573137576 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 40 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 41, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 00:54:03:783.760 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 00:54:03:783.735 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: RAC_Connect status: 0 0xd43ca0 2025-02-17 Mon 00:54:03:783.786 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:783.776 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 00:54:03:783.798 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:783.819 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: RAC_Connect status: 0 0xd49260 2025-02-17 Mon 00:54:03:783.827 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:783.822 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: SANDESH: Send FAILED: 1739753643783420 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-17 Mon 00:54:03:783.852 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 00:54:03:783.839 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:783.864 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 00:54:03:783.882 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: RAC_Connect status: 0 0xd44870 2025-02-17 Mon 00:54:03:783.905 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:783.913 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:783.923 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: RAC_Connect status: 0 0xd3d1d0 2025-02-17 Mon 00:54:03:783.928 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:783.935 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 00:54:03:783.937 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:783.957 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: RAC_Connect status: 0 0xd49ba0 2025-02-17 Mon 00:54:03:783.962 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:783.972 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:783.981 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: RAC_Connect status: 0 0xd44340 2025-02-17 Mon 00:54:03:783.986 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:783.996 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:784.006 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: RAC_Connect status: 0 0xd4a4e0 2025-02-17 Mon 00:54:03:784.012 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: Connected to REDIS... 2025-02-17 Mon 00:54:03:784.026 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 00:54:03:784.053 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: SANDESH: Send FAILED: 1739753643783560 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739753573137576 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 42 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 44, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 00:54:03:784.070 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 00:54:03:784.166 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 00:54:03:784.209 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: SANDESH: Send FAILED: 1739753643784096 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739753643783498 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739753643783932 ] max_sm_queue_count = 5 ] 2025-02-17 Mon 00:54:03:784.216 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 00:54:03:784.224 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 00:54:03:784.484 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:784.498 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: SANDESH: Send FAILED: 1739753643784153 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739753573137576 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 54 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 56, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 10, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 00:54:03:784.566 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 00:54:03:784.544 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:784.629 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:784.639 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:784.652 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:784.668 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:784.680 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:784.704 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:784.723 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:784.732 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:784.812 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:784.851 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243055380224, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 00:54:03:784.868 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:784.911 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243055380224, Pid 1]: SANDESH: Send FAILED: 1739753643784787 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 00:54:03:784.933 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243055380224, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 00:54:03:784.949 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243055380224, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 00:54:03:784.936 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:784.971 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243055380224, Pid 1]: SANDESH: Send FAILED: 1739753643784860 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.56:6379::Down.10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 00:54:03:784.985 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243055380224, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 00:54:03:785.020 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:785.054 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:785.078 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:785.088 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:785.100 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:785.109 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:785.121 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:785.130 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:785.142 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:785.151 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:785.187 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:785.225 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:785.233 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 00:54:03:785.242 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:785.274 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: SANDESH: Send FAILED: 1739753643785207 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.56:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 00:54:03:785.282 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:785.540 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:785.552 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:785.566 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:785.575 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:785.586 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:785.595 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:785.607 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:785.616 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:785.628 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:785.637 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:785.648 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 00:54:03:785.293 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 00:54:03:785.714 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 00:54:03:785.760 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:785.786 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:785.799 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:785.818 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:785.831 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:785.843 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 00:54:03:785.771 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243084760832, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 00:54:03:785.932 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243084760832, Pid 1]: SANDESH: Send FAILED: 1739753643785740 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 00:54:03:785.940 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243084760832, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 00:54:04:783.675 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243183904512, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-17 Mon 00:54:04:783.938 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243183904512, Pid 1]: Connect : Start Connect timer 10.0.0.56:8086 2025-02-17 Mon 00:54:04:784.085 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243183904512, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 00:54:04:784.120 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243183904512, Pid 1]: SANDESH: Send FAILED: 1739753644783762 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 00:54:04:784.130 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243183904512, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 00:54:04:784.140 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243183904512, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 00:54:04:784.265 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243183904512, Pid 1]: SANDESH: Send FAILED: 1739753644784072 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739753573137576 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 99 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 101, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 00:54:04:784.280 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243183904512, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 00:54:04:786.870 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.53:51022::10.0.0.56:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-17 Mon 00:54:04:786.901 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: 10.0.0.53:51022::10.0.0.56:8086(-1) OnSessionEvent TCP Connected 2025-02-17 Mon 00:54:04:786.909 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: 10.0.0.53:51022::10.0.0.56:8086(-1):EvTcpConnected 2025-02-17 Mon 00:54:04:786.970 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243084760832, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-17 Mon 00:54:04:787.000 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243084760832, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-17 Mon 00:54:04:787.041 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243084760832, Pid 1]: ClientInit 2025-02-17 Mon 00:54:04:787.063 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243084760832, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2738-2.:contrail-query-engine:0:Database count 1 2025-02-17 Mon 00:54:04:787.163 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243084760832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 00:54:04:787.173 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243084760832, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 00:54:04:787.196 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243084760832, Pid 1]: SANDESH: Send FAILED: 1739753644787033 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 00:54:04:787.204 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243084760832, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 00:54:04:787.213 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243084760832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 00:54:04:787.219 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243084760832, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-17 Mon 00:54:04:787.230 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243084760832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 00:54:04:787.246 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243084760832, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-17 Mon 00:54:04:787.373 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243084760832, Pid 1]: SANDESH: Send FAILED: 1739753644787150 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739753573137576 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 99 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 102, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 00:54:04:787.388 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243084760832, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-17 Mon 00:54:04:791.665 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: OnMessage control in state: ClientInit 2025-02-17 Mon 00:54:04:791.725 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-17 Mon 00:54:04:791.736 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-17 Mon 00:54:04:791.925 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: Received Ctrl Message with size 0 2025-02-17 Mon 00:54:04:792.073 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739753573137576 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739753643783498 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739753643783932 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 99 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 102, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 5 proxy seq 10 2025-02-17 Mon 00:54:04:792.101 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-17 Mon 00:54:04:792.127 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2025-02-17 Mon 00:54:04:792.141 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-17 Mon 00:54:04:792.149 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-17 Mon 00:54:04:792.157 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-17 Mon 00:54:04:792.185 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: Established 2025-02-17 Mon 00:54:14:478.801 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243076364032, Pid 1]: OnMessage control in state: Established 2025-02-17 Mon 00:54:14:478.864 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243076364032, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2025-02-17 Mon 00:54:14:478.875 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243076364032, Pid 1]: Established : EvSandeshMessageRecv 2025-02-17 Mon 00:54:14:478.934 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243076364032, Pid 1]: Received Ctrl Message with size 3 2025-02-17 Mon 00:54:14:478.955 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243076364032, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 11, total = 0 2025-02-17 Mon 00:54:14:478.968 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243076364032, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 13, total = 0 2025-02-17 Mon 00:54:14:478.979 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243076364032, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2025-02-17 Mon 00:54:14:478.987 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243076364032, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-17 Mon 00:54:33:784.485 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242149431040, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-17 Mon 00:55:33:785.062 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-17 Mon 00:56:33:785.626 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242149431040, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-17 Mon 00:57:12:278.171 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.53:8091::10.0.0.53:39658 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-02-17 Mon 00:57:33:787.378 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243080562432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-17 Mon 00:57:34:164.950 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243331774592, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.53:8091::10.0.0.53:54164 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-02-17 Mon 00:58:33:788.096 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243183904512, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-17 Mon 00:59:33:788.700 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-17 Mon 01:00:33:789.317 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-17 Mon 01:01:33:790.904 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242145232640, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-17 Mon 01:02:33:791.527 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-17 Mon 01:03:33:792.079 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243055380224, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-02-17 Mon 01:04:33:792.641 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243183904512, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-02-17 Mon 01:05:33:793.227 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243055380224, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-02-17 Mon 01:06:33:795.765 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242153629440, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-02-17 Mon 01:07:33:797.763 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-02-17 Mon 01:08:33:798.289 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243175507712, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-02-17 Mon 01:09:33:798.899 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243171309312, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-02-17 Mon 01:10:33:799.462 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243175507712, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-02-17 Mon 01:11:33:800.040 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-02-17 Mon 01:12:33:800.664 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243171309312, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-02-17 Mon 01:13:33:801.198 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242149431040, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-02-17 Mon 01:14:33:801.775 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243171309312, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-02-17 Mon 01:15:33:802.335 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243055380224, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-02-17 Mon 01:16:33:802.867 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243183904512, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-02-17 Mon 01:17:33:803.440 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243076364032, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-02-17 Mon 01:18:33:804.042 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242149431040, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-02-17 Mon 01:19:33:804.572 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243080562432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2025-02-17 Mon 01:20:33:805.130 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242149431040, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2025-02-17 Mon 01:21:33:805.663 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242153629440, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2025-02-17 Mon 01:22:33:806.270 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243188102912, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0 2025-02-17 Mon 01:23:33:806.819 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243080562432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 30 without seqno , total = 0 2025-02-17 Mon 01:24:33:807.366 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243175507712, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 31 without seqno , total = 0 2025-02-17 Mon 01:25:33:807.945 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243175507712, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 32 without seqno , total = 0 2025-02-17 Mon 01:26:33:808.501 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243080562432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 33 without seqno , total = 0 2025-02-17 Mon 01:27:33:809.094 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243084760832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 34 without seqno , total = 0 2025-02-17 Mon 01:28:33:809.729 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243080562432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 35 without seqno , total = 0 2025-02-17 Mon 01:29:33:810.267 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243188102912, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 36 without seqno , total = 0 2025-02-17 Mon 01:30:33:810.817 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243188102912, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 37 without seqno , total = 0 2025-02-17 Mon 01:31:33:811.974 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243076364032, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 38 without seqno , total = 0 2025-02-17 Mon 01:32:33:812.540 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242145232640, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 39 without seqno , total = 0 2025-02-17 Mon 01:33:33:813.542 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243080562432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 40 without seqno , total = 0 2025-02-17 Mon 01:34:33:814.042 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243076364032, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 41 without seqno , total = 0 2025-02-17 Mon 01:35:33:814.590 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243055380224, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 42 without seqno , total = 0 2025-02-17 Mon 01:36:33:815.966 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243171309312, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 43 without seqno , total = 0 2025-02-17 Mon 01:37:33:816.594 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243055380224, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 44 without seqno , total = 0 2025-02-17 Mon 01:38:33:817.184 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242145232640, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 45 without seqno , total = 0 2025-02-17 Mon 01:39:33:817.664 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243179706112, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 46 without seqno , total = 0 2025-02-17 Mon 01:40:33:818.335 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242145232640, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 47 without seqno , total = 0 2025-02-17 Mon 01:41:33:819.079 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243059578624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 48 without seqno , total = 0 2025-02-17 Mon 01:42:33:819.628 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242149431040, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 49 without seqno , total = 0 2025-02-17 Mon 01:43:33:820.250 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243080562432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 50 without seqno , total = 0 2025-02-17 Mon 01:44:33:820.903 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243171309312, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 51 without seqno , total = 0 2025-02-17 Mon 01:45:33:821.467 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 52 without seqno , total = 0 2025-02-17 Mon 01:46:33:822.548 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243067971328, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 53 without seqno , total = 0 2025-02-17 Mon 01:47:33:823.076 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243183904512, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 54 without seqno , total = 0 2025-02-17 Mon 01:48:33:823.680 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243175507712, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 55 without seqno , total = 0 2025-02-17 Mon 01:49:33:824.234 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243183904512, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 56 without seqno , total = 0 2025-02-17 Mon 01:50:33:824.811 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243059578624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 57 without seqno , total = 0 2025-02-17 Mon 01:51:33:825.413 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243059578624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 58 without seqno , total = 0 2025-02-17 Mon 01:52:33:825.951 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140242157827840, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 59 without seqno , total = 0 2025-02-17 Mon 01:53:23:822.599 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243183904512, Pid 1]: Processing scm::EvStop in state Established 2025-02-17 Mon 01:53:23:822.680 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243183904512, Pid 1]: Idle 2025-02-17 Mon 01:53:23:822.793 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243183904512, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 01:53:23:822.804 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243183904512, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:53:23:822.963 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243183904512, Pid 1]: SANDESH: Send FAILED: 1739757203822779 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739753573137576 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 2874993 calls = 301 average_bytes = 9551 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 60 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 99 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 238 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 104, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 302, ] ] ] 2025-02-17 Mon 01:53:23:822.987 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140243183904512, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:53:38:960.179 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: http-server-port 8091 2025-02-17 Mon 01:53:38:960.264 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Max-tasks 16 2025-02-17 Mon 01:53:38:960.272 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Max-slice 100 2025-02-17 Mon 01:53:38:960.278 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Collectors 10.0.0.56:8086 2025-02-17 Mon 01:53:38:960.283 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Collectors 10.0.0.53:8086 2025-02-17 Mon 01:53:38:960.287 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Collectors 10.0.0.22:8086 2025-02-17 Mon 01:53:38:960.306 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: SANDESH: ROLE : Generator 2025-02-17 Mon 01:53:38:960.312 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-17 Mon 01:53:38:994.102 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2738-2. 2025-02-17 Mon 01:53:38:994.109 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-17 Mon 01:53:38:994.115 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-17 Mon 01:53:38:994.122 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-17 Mon 01:53:38:994.128 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-17 Mon 01:53:38:994.762 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: HTTP Introspect Init 2025-02-17 Mon 01:53:38:994.887 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-17 Mon 01:53:38:994.959 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Sandesh Http Server Port: 8091 2025-02-17 Mon 01:53:38:994.992 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-17 Mon 01:53:38:995.007 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-17 Mon 01:53:38:995.084 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Idle 2025-02-17 Mon 01:53:38:995.249 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: SANDESH: No Client: 1739757218995176 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 01:53:38:995.374 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-17 Mon 01:53:38:995.390 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-17 Mon 01:53:38:995.404 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Cassandra Servers: 10.0.0.56:9042 10.0.0.53:9042 10.0.0.22:9042 2025-02-17 Mon 01:53:38:999.805 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: Processing scm::EvStart in state Idle 2025-02-17 Mon 01:53:39:008.273 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: Disconnect 2025-02-17 Mon 01:53:39:008.416 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-17 Mon 01:53:39:008.633 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: Connect : Start Connect timer 10.0.0.22:8086 2025-02-17 Mon 01:53:39:008.706 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:53:39:008.727 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: SANDESH: Send FAILED: 1739757219008245 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-17 Mon 01:53:39:008.744 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:53:39:008.754 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:53:39:008.855 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: SANDESH: Send FAILED: 1739757219008402 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 01:53:39:008.867 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:53:39:008.877 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:53:39:009.601 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: SANDESH: Send FAILED: 1739757219008482 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 01:53:39:009.613 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:53:39:009.625 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:53:39:009.641 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: SANDESH: Send FAILED: 1739757219008514 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-17 Mon 01:53:39:009.648 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:53:39:009.660 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:53:39:009.723 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: SANDESH: Send FAILED: 1739757219008694 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 01:53:39:009.732 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:53:39:197.699 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-17 Mon 01:54:09:212.555 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:54:09:212.623 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: SANDESH: Send FAILED: 1739757249212466 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-17 Mon 01:54:09:212.639 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:54:09:212.654 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:54:09:213.152 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:54:09:213.190 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:54:09:213.208 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:54:09:213.222 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:54:09:213.237 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c28830 2025-02-17 Mon 01:54:09:213.243 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:54:09:213.259 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:54:09:213.270 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c29400 2025-02-17 Mon 01:54:09:213.275 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:54:09:213.282 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:54:09:213.293 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:54:09:213.306 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c24da0 2025-02-17 Mon 01:54:09:213.311 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:54:09:213.319 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:54:09:213.329 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:54:09:213.342 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:54:09:213.355 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:54:09:213.367 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c29b50 2025-02-17 Mon 01:54:09:213.372 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:54:09:213.380 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:54:09:213.397 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:54:09:213.409 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c2a470 2025-02-17 Mon 01:54:09:213.414 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:54:09:213.423 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:54:09:215.378 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.53:53086::10.0.0.22:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-17 Mon 01:54:09:215.400 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: 10.0.0.53:53086::10.0.0.22:8086(-1) OnSessionEvent TCP Connected 2025-02-17 Mon 01:54:09:215.407 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: 10.0.0.53:53086::10.0.0.22:8086(-1):EvTcpConnected 2025-02-17 Mon 01:54:09:215.464 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-17 Mon 01:54:09:215.490 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-17 Mon 01:54:09:215.535 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: ClientInit 2025-02-17 Mon 01:54:09:215.552 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2738-2.:contrail-query-engine:0:Database count 1 2025-02-17 Mon 01:54:09:215.643 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:54:09:215.653 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 01:54:09:215.684 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: SANDESH: Send FAILED: 1739757249215527 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-17 Mon 01:54:09:215.692 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 01:54:09:215.701 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:54:09:215.707 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-17 Mon 01:54:09:215.719 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:54:09:215.725 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-17 Mon 01:54:09:215.843 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: SANDESH: Send FAILED: 1739757249215632 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 8, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 01:54:09:215.864 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-17 Mon 01:54:09:216.035 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:54:09:216.046 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-17 Mon 01:54:09:216.105 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: SANDESH: Send FAILED: 1739757249215910 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = ClientInit state_since = 1739757249215533 last_state = ClientInit last_event = scm::EvSandeshSend last_event_at = 1739757249215717 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 5 ] 2025-02-17 Mon 01:54:09:216.115 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-17 Mon 01:54:09:216.123 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:54:09:216.130 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-17 Mon 01:54:09:216.229 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: SANDESH: Send FAILED: 1739757249215953 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 8, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 8, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 01:54:09:216.245 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700117370624, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-17 Mon 01:54:09:216.559 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:54:09:216.599 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:54:09:216.615 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:54:09:216.624 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:54:09:216.636 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:54:09:216.676 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:54:09:216.690 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:54:09:216.699 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:54:09:216.711 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:54:09:216.720 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:54:09:216.786 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:54:09:216.801 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:54:09:216.817 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:54:09:216.837 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:54:09:216.852 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:54:09:216.867 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:54:09:216.819 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 01:54:09:217.012 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: SANDESH: Send FAILED: 1739757249216761 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:54:09:217.025 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 01:54:09:217.034 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:54:09:217.042 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 01:54:09:217.060 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: SANDESH: Send FAILED: 1739757249216811 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.53:6379::Down.10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 01:54:09:217.067 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 01:54:09:217.439 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: OnMessage control in state: ClientInit 2025-02-17 Mon 01:54:09:217.492 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-17 Mon 01:54:09:217.521 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-17 Mon 01:54:09:217.551 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Received Ctrl Message with size 0 2025-02-17 Mon 01:54:09:217.716 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = ClientInit state_since = 1739757249215533 last_state = ClientInit last_event = scm::EvSandeshSend last_event_at = 1739757249215717 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 8, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 8, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 5 proxy seq 7 2025-02-17 Mon 01:54:09:217.745 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-17 Mon 01:54:09:217.769 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.53:6379::Down.10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 6 2025-02-17 Mon 01:54:09:217.780 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-17 Mon 01:54:09:217.788 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-17 Mon 01:54:09:217.795 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-17 Mon 01:54:09:217.823 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Established 2025-02-17 Mon 01:54:14:214.014 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:54:14:214.141 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:54:14:214.186 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:54:14:214.224 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:54:14:214.259 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:54:14:214.297 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:54:14:214.327 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:54:14:214.355 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:54:14:214.383 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:54:14:214.412 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:54:14:214.456 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c2ad70 2025-02-17 Mon 01:54:14:214.463 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:54:14:214.484 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:54:14:214.497 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c2b6b0 2025-02-17 Mon 01:54:14:214.503 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:54:14:214.512 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:54:14:214.520 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c2bff0 2025-02-17 Mon 01:54:14:214.525 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:54:14:214.533 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:54:14:214.541 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c2c930 2025-02-17 Mon 01:54:14:214.546 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:54:14:214.554 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:54:14:214.564 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c2d270 2025-02-17 Mon 01:54:14:214.569 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:54:14:214.577 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:54:14:214.622 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c2dbb0 2025-02-17 Mon 01:54:14:214.630 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:54:14:214.642 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:54:14:214.666 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c2e4f0 2025-02-17 Mon 01:54:14:214.674 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:54:14:214.685 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:54:14:214.732 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c2ee30 2025-02-17 Mon 01:54:14:214.738 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:54:14:214.749 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:54:14:214.763 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c2f770 2025-02-17 Mon 01:54:14:214.768 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:54:14:214.776 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:54:14:214.784 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c300b0 2025-02-17 Mon 01:54:14:214.791 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:54:14:214.799 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:54:14:215.104 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:54:14:215.124 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:54:14:215.138 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:54:14:215.148 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:54:14:215.161 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:54:14:215.170 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:54:14:215.184 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:54:14:215.193 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:54:14:215.206 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:54:14:215.219 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:54:14:215.232 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:54:14:215.243 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:54:14:215.275 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:54:14:215.330 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:54:14:215.358 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:54:14:215.372 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:54:14:215.386 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:54:14:215.398 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:54:14:215.419 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:54:14:215.428 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:54:14:215.441 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:54:14:215.450 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:54:14:215.474 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:54:14:215.484 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:54:14:215.500 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:54:14:215.513 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:54:14:215.526 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:54:14:215.538 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:54:14:215.548 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:54:14:215.562 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:54:39:216.732 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700125767424, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-17 Mon 01:55:39:217.313 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700142561024, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-17 Mon 01:56:24:305.186 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Disconnect status: -1 0x2c2d270 2025-02-17 Mon 01:56:24:305.249 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:305.427 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Disconnect status: -1 0x2c2c930 2025-02-17 Mon 01:56:24:305.445 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:305.477 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Disconnect status: -1 0x2c2bff0 2025-02-17 Mon 01:56:24:305.489 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:305.514 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Disconnect status: -1 0x2c2b6b0 2025-02-17 Mon 01:56:24:305.525 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:305.553 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-17 Mon 01:56:24:305.560 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Disconnect status: -1 0x2c2ad70 2025-02-17 Mon 01:56:24:305.569 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:307.212 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:56:24:307.303 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:56:24:307.322 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:56:24:307.336 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:56:24:307.348 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:56:24:309.130 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Disconnect status: -1 0x2c2a470 2025-02-17 Mon 01:56:24:309.160 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:309.198 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Disconnect status: -1 0x2c29b50 2025-02-17 Mon 01:56:24:309.209 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:309.239 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Disconnect status: -1 0x2c24da0 2025-02-17 Mon 01:56:24:309.249 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:309.270 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Disconnect status: -1 0x2c29400 2025-02-17 Mon 01:56:24:309.279 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:309.304 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-17 Mon 01:56:24:309.313 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Disconnect status: -1 0x2c28830 2025-02-17 Mon 01:56:24:309.325 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:310.079 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:56:24:310.115 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:56:24:310.130 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:56:24:310.144 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:56:24:310.157 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:56:24:437.403 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Disconnect status: -1 0x2c300b0 2025-02-17 Mon 01:56:24:437.460 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:437.607 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Disconnect status: -1 0x2c2f770 2025-02-17 Mon 01:56:24:437.623 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:437.644 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Disconnect status: -1 0x2c2ee30 2025-02-17 Mon 01:56:24:437.655 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:437.674 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Disconnect status: -1 0x2c2e4f0 2025-02-17 Mon 01:56:24:437.683 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:437.720 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-02-17 Mon 01:56:24:437.748 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Disconnect status: -1 0x2c2dbb0 2025-02-17 Mon 01:56:24:437.757 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-02-17 Mon 01:56:24:438.283 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:56:24:438.386 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:56:24:438.407 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:56:24:438.475 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:56:24:438.491 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:56:24:451.494 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.53:53086::10.0.0.22:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-17 Mon 01:56:24:451.546 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: 10.0.0.53:53086::10.0.0.22:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-17 Mon 01:56:24:451.554 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: 10.0.0.53:53086::10.0.0.22:8086(-1):EvTcpClose 2025-02-17 Mon 01:56:24:451.638 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700155819776, Pid 1]: Processing scm::EvTcpClose in state Established 2025-02-17 Mon 01:56:24:451.668 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700155819776, Pid 1]: Established : EvTcpClose 2025-02-17 Mon 01:56:24:451.905 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700155819776, Pid 1]: Connect : Start Connect timer 10.0.0.56:8086 2025-02-17 Mon 01:56:24:452.006 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700155819776, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:24:452.154 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700155819776, Pid 1]: SANDESH: Send FAILED: 1739757384451764 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = cn-jenkins-deploy-platform-ansible-os-2738-3. collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 151151 calls = 20 average_bytes = 7557 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 82, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 12, _iter59->first = sent _iter59->second = 22, ] ] ] 2025-02-17 Mon 01:56:24:452.177 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700155819776, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:24:452.188 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700155819776, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:24:452.211 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700155819776, Pid 1]: SANDESH: Send FAILED: 1739757384451796 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:24:452.219 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700155819776, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:24:452.228 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700155819776, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2025-02-17 Mon 01:56:24:452.266 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700155819776, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:24:452.415 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700155819776, Pid 1]: SANDESH: Send FAILED: 1739757384451993 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 151151 calls = 20 average_bytes = 7557 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 82, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 12, _iter59->first = sent _iter59->second = 22, ] ] ] 2025-02-17 Mon 01:56:24:452.434 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700155819776, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:24:455.129 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.53:57728::10.0.0.56:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-17 Mon 01:56:24:455.153 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: 10.0.0.53:57728::10.0.0.56:8086(-1) OnSessionEvent TCP Connected 2025-02-17 Mon 01:56:24:455.163 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: 10.0.0.53:57728::10.0.0.56:8086(-1):EvTcpConnected 2025-02-17 Mon 01:56:24:455.639 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700142561024, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-17 Mon 01:56:24:455.683 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700142561024, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-17 Mon 01:56:24:455.718 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700142561024, Pid 1]: ClientInit 2025-02-17 Mon 01:56:24:455.756 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700142561024, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2738-2.:contrail-query-engine:0:Database count 2 2025-02-17 Mon 01:56:24:455.863 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700142561024, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:24:455.874 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700142561024, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 01:56:24:455.897 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700142561024, Pid 1]: SANDESH: Send FAILED: 1739757384455711 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:24:455.913 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700142561024, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 01:56:24:455.936 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700142561024, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:24:455.946 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700142561024, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-17 Mon 01:56:24:455.958 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700142561024, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:24:455.964 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700142561024, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-17 Mon 01:56:24:456.198 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700142561024, Pid 1]: SANDESH: Send FAILED: 1739757384455846 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 151151 calls = 20 average_bytes = 7557 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 15, _iter59->first = sent _iter59->second = 22, ] ] ] 2025-02-17 Mon 01:56:24:456.241 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700142561024, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-17 Mon 01:56:24:477.579 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: 10.0.0.53:57728::10.0.0.56:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-17 Mon 01:56:24:490.148 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: 10.0.0.53:57728::10.0.0.56:8086(-1):EvTcpClose 2025-02-17 Mon 01:56:24:490.225 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-17 Mon 01:56:24:490.253 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: ClientInit : EvTcpClose 2025-02-17 Mon 01:56:24:490.396 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Return to idle with 5000 2025-02-17 Mon 01:56:24:490.406 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Idle 2025-02-17 Mon 01:56:24:490.461 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:24:490.605 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: SANDESH: Send FAILED: 1739757384490350 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 152724 calls = 21 average_bytes = 7272 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 17, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-17 Mon 01:56:24:490.623 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:24:490.633 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:24:490.656 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: SANDESH: Send FAILED: 1739757384490383 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:24:490.666 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:24:490.674 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 01:56:24:490.709 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:24:490.878 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: SANDESH: Send FAILED: 1739757384490452 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 152724 calls = 21 average_bytes = 7272 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 17, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-17 Mon 01:56:24:490.894 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:29:307.302 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:56:29:307.481 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:56:29:307.514 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:56:29:307.556 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:56:29:307.600 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:56:29:307.644 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:56:29:307.677 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:56:29:307.694 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:56:29:307.706 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:56:29:307.718 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:56:29:310.129 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:56:29:310.184 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:56:29:310.210 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:56:29:310.234 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:56:29:310.259 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:56:29:310.556 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:56:29:310.604 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:56:29:310.619 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:56:29:310.632 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:56:29:310.660 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:56:29:438.360 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:56:29:438.521 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:56:29:438.555 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:56:29:438.584 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:56:29:438.612 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:56:29:438.824 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:56:29:438.879 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:56:29:438.900 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:56:29:438.930 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:56:29:438.944 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:56:29:492.635 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-17 Mon 01:56:29:492.905 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: Connect : Start Connect timer 10.0.0.53:8086 2025-02-17 Mon 01:56:29:493.056 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:29:493.088 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: SANDESH: Send FAILED: 1739757389492719 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.53:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:29:493.097 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:29:493.107 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:29:493.250 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: SANDESH: Send FAILED: 1739757389493039 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 152724 calls = 21 average_bytes = 7272 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-17 Mon 01:56:29:493.278 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:29:495.192 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.53:51524::10.0.0.53:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-17 Mon 01:56:29:495.222 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: 10.0.0.53:51524::10.0.0.53:8086(-1) OnSessionEvent TCP Connected 2025-02-17 Mon 01:56:29:495.231 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: 10.0.0.53:51524::10.0.0.53:8086(-1):EvTcpConnected 2025-02-17 Mon 01:56:29:495.280 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-17 Mon 01:56:29:495.304 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-17 Mon 01:56:29:495.337 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: ClientInit 2025-02-17 Mon 01:56:29:495.404 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2738-2.:contrail-query-engine:0:Database count 3 2025-02-17 Mon 01:56:29:495.542 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:29:495.554 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 01:56:29:495.578 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: SANDESH: Send FAILED: 1739757389495330 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.53:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:29:495.586 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 01:56:29:495.595 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:29:495.602 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-17 Mon 01:56:29:495.613 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:29:495.620 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-17 Mon 01:56:29:495.784 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: SANDESH: Send FAILED: 1739757389495530 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 152724 calls = 21 average_bytes = 7272 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 22, _iter59->first = sent _iter59->second = 23, ] ] ] 2025-02-17 Mon 01:56:29:495.805 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-17 Mon 01:56:31:495.516 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: 10.0.0.53:8086 EvConnectTimerExpired in state ClientInit 2025-02-17 Mon 01:56:31:495.588 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: Processing scm::EvConnectTimerExpired in state ClientInit 2025-02-17 Mon 01:56:31:495.602 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: ClientInit : EvConnectTimerExpired 2025-02-17 Mon 01:56:31:495.826 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: Return to idle with 5000 2025-02-17 Mon 01:56:31:495.838 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: Idle 2025-02-17 Mon 01:56:31:495.907 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:31:496.081 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: SANDESH: Send FAILED: 1739757391495690 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 154297 calls = 22 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 24, _iter59->first = sent _iter59->second = 24, ] ] ] 2025-02-17 Mon 01:56:31:496.114 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:31:496.125 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:31:496.150 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: SANDESH: Send FAILED: 1739757391495722 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.53:8086, ] ] status = Down description = ClientInit : EvConnectTimerExpired ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:31:496.158 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:31:496.167 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 01:56:31:496.194 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:31:496.349 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: SANDESH: Send FAILED: 1739757391495893 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 154297 calls = 22 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 24, _iter59->first = sent _iter59->second = 24, ] ] ] 2025-02-17 Mon 01:56:31:496.367 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:34:307.563 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:56:34:307.742 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:56:34:307.772 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:56:34:307.807 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:56:34:307.840 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:56:34:307.876 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:56:34:307.891 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:56:34:307.904 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:56:34:307.913 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:56:34:307.962 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:56:34:310.604 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:56:34:310.679 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:56:34:310.722 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:56:34:310.747 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:56:34:310.773 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:56:34:310.960 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:56:34:310.980 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:56:34:311.044 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:56:34:311.076 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:56:34:311.091 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:56:34:438.892 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:56:34:439.087 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:56:34:439.125 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:56:34:439.153 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:56:34:439.186 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:56:34:439.379 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:56:34:439.410 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:56:34:439.423 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:56:34:439.435 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:56:34:439.448 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:56:36:496.009 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-17 Mon 01:56:36:496.285 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Connect : Start Connect timer 10.0.0.22:8086 2025-02-17 Mon 01:56:36:496.419 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:36:496.449 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: SANDESH: Send FAILED: 1739757396496116 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:36:496.470 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:36:496.479 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:36:496.651 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: SANDESH: Send FAILED: 1739757396496404 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 154297 calls = 22 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 27, _iter59->first = sent _iter59->second = 24, ] ] ] 2025-02-17 Mon 01:56:36:496.670 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:36:498.551 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.53:42714::10.0.0.22:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-17 Mon 01:56:36:498.580 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: 10.0.0.53:42714::10.0.0.22:8086(-1) OnSessionEvent TCP Connected 2025-02-17 Mon 01:56:36:498.588 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: 10.0.0.53:42714::10.0.0.22:8086(-1):EvTcpConnected 2025-02-17 Mon 01:56:36:498.647 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-17 Mon 01:56:36:498.687 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-17 Mon 01:56:36:498.725 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: ClientInit 2025-02-17 Mon 01:56:36:498.741 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2738-2.:contrail-query-engine:0:Database count 4 2025-02-17 Mon 01:56:36:498.822 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:36:498.831 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 01:56:36:498.854 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: SANDESH: Send FAILED: 1739757396498716 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:36:498.862 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 01:56:36:498.871 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:36:498.877 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-17 Mon 01:56:36:498.888 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:56:36:498.894 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-17 Mon 01:56:36:499.051 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: SANDESH: Send FAILED: 1739757396498812 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 154297 calls = 22 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 29, _iter59->first = sent _iter59->second = 24, ] ] ] 2025-02-17 Mon 01:56:36:499.071 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-17 Mon 01:56:37:055.769 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.53:42714::10.0.0.22:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2025-02-17 Mon 01:56:37:055.809 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.53:42714::10.0.0.22:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-17 Mon 01:56:37:055.831 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: 10.0.0.53:42714::10.0.0.22:8086(-1) OnSessionEvent TCP Connection Closed 2025-02-17 Mon 01:56:37:055.839 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: 10.0.0.53:42714::10.0.0.22:8086(-1):EvTcpClose 2025-02-17 Mon 01:56:37:056.445 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700168414976, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-02-17 Mon 01:56:37:056.495 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700168414976, Pid 1]: ClientInit : EvTcpClose 2025-02-17 Mon 01:56:37:056.689 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700168414976, Pid 1]: Return to idle with 5000 2025-02-17 Mon 01:56:37:056.703 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700168414976, Pid 1]: Idle 2025-02-17 Mon 01:56:37:056.771 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700168414976, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:37:056.939 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700168414976, Pid 1]: SANDESH: Send FAILED: 1739757397056614 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 31, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:56:37:056.963 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700168414976, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:37:056.974 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700168414976, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:37:056.996 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700168414976, Pid 1]: SANDESH: Send FAILED: 1739757397056671 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:37:057.003 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700168414976, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:37:057.012 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700168414976, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 01:56:37:057.039 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700168414976, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:37:057.170 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700168414976, Pid 1]: SANDESH: Send FAILED: 1739757397056760 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 31, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:56:37:057.186 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700168414976, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:39:218.107 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700125767424, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-17 Mon 01:56:39:218.194 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:39:218.293 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: SANDESH: Send FAILED: 1739757399217965 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 59 dequeues = 59 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 76 dequeues = 76 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739757397056701 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739757397057037 ] max_sm_queue_count = 5 ] 2025-02-17 Mon 01:56:39:218.308 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:39:218.319 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:39:218.485 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: SANDESH: Send FAILED: 1739757399218085 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 34, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:56:39:218.518 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:39:307.814 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:56:39:308.016 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:56:39:308.051 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:56:39:308.089 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:56:39:308.119 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:56:39:308.147 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:56:39:308.182 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:56:39:308.197 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:56:39:308.210 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:56:39:308.222 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:56:39:311.003 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:56:39:311.092 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:56:39:311.139 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:56:39:311.176 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:56:39:311.210 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:56:39:311.391 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:56:39:311.485 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:56:39:311.506 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:56:39:311.520 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:56:39:311.595 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:56:39:439.467 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:56:39:439.626 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:56:39:439.678 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:56:39:439.717 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:56:39:439.746 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:56:39:439.901 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:56:39:439.967 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:56:39:439.984 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:56:39:440.146 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:56:39:440.171 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:56:42:056.865 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-17 Mon 01:56:42:057.123 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Connect : Start Connect timer 10.0.0.56:8086 2025-02-17 Mon 01:56:42:057.228 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:42:057.257 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: SANDESH: Send FAILED: 1739757402056947 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:42:057.277 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:42:057.286 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:42:057.516 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: SANDESH: Send FAILED: 1739757402057215 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 36, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:56:42:057.537 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:42:057.616 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-17 Mon 01:56:42:057.625 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: -(-1):EvTcpConnectFail 2025-02-17 Mon 01:56:42:057.685 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-17 Mon 01:56:42:057.728 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Connect : EvTcpConnectFail 2025-02-17 Mon 01:56:42:057.927 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-17 Mon 01:56:42:057.969 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Idle 2025-02-17 Mon 01:56:42:058.032 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:42:058.179 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: SANDESH: Send FAILED: 1739757402057845 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 38, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:56:42:058.199 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:42:058.210 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:42:058.232 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: SANDESH: Send FAILED: 1739757402057887 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:42:058.243 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:42:058.252 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 01:56:42:058.279 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:42:058.428 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: SANDESH: Send FAILED: 1739757402058022 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 38, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:56:42:058.445 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:44:308.106 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:56:44:308.268 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:56:44:308.302 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:56:44:308.368 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:56:44:308.413 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:56:44:308.444 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:56:44:308.485 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:56:44:308.504 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:56:44:308.517 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:56:44:308.531 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:56:44:311.434 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:56:44:311.518 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:56:44:311.560 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:56:44:311.589 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:56:44:311.624 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:56:44:311.780 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:56:44:311.831 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:56:44:311.856 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:56:44:311.926 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:56:44:311.976 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:56:44:440.078 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:56:44:440.235 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:56:44:440.276 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:56:44:440.432 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:56:44:440.479 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:56:44:440.520 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:56:44:440.546 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:56:44:440.560 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:56:44:440.604 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:56:44:440.639 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:56:49:308.377 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:56:49:308.577 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:56:49:308.607 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:56:49:308.642 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:56:49:308.670 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:56:49:308.708 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:56:49:308.743 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:56:49:308.758 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:56:49:308.771 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:56:49:308.783 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:56:49:311.821 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:56:49:311.904 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:56:49:311.955 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:56:49:311.988 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:56:49:312.017 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:56:49:312.284 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:56:49:312.343 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:56:49:312.360 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:56:49:312.499 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:56:49:312.514 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:56:49:440.592 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:56:49:440.746 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:56:49:440.784 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:56:49:440.810 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:56:49:440.836 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:56:49:441.046 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:56:49:441.081 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:56:49:441.096 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:56:49:441.108 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:56:49:441.134 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:56:50:058.518 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-17 Mon 01:56:50:058.737 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Connect : Start Connect timer 10.0.0.53:8086 2025-02-17 Mon 01:56:50:058.848 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:50:059.065 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: SANDESH: Send FAILED: 1739757410058586 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.53:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:50:059.079 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:50:059.091 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:56:50:059.258 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: SANDESH: Send FAILED: 1739757410058834 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 41, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:56:50:059.280 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:50:059.581 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-17 Mon 01:56:50:059.590 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: -(-1):EvTcpConnectFail 2025-02-17 Mon 01:56:50:059.885 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-17 Mon 01:56:50:059.896 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Connect : EvTcpConnectFail 2025-02-17 Mon 01:56:50:060.058 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-17 Mon 01:56:50:060.105 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Idle 2025-02-17 Mon 01:56:50:060.198 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:50:060.369 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: SANDESH: Send FAILED: 1739757410059970 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 43, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:56:50:060.394 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:50:060.405 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:50:060.427 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: SANDESH: Send FAILED: 1739757410059992 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:56:50:060.434 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:56:50:060.443 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 01:56:50:060.471 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:56:50:060.600 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: SANDESH: Send FAILED: 1739757410060184 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 43, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:56:50:060.617 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:56:54:308.654 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:56:54:308.853 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:56:54:308.880 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:56:54:308.939 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:56:54:308.972 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:56:54:309.001 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:56:54:309.053 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:56:54:309.070 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:56:54:309.083 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:56:54:309.095 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:56:54:312.359 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:56:54:312.455 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:56:54:312.488 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:56:54:312.580 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:56:54:312.621 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:56:54:312.776 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:56:54:312.827 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:56:54:312.854 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:56:54:312.934 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:56:54:312.951 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:56:54:441.126 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:56:54:441.308 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:56:54:441.382 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:56:54:441.424 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:56:54:441.460 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:56:54:441.621 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:56:54:441.654 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:56:54:441.669 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:56:54:441.682 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:56:54:441.694 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:56:59:308.955 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:56:59:309.167 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:56:59:309.198 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:56:59:309.232 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:56:59:309.262 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:56:59:309.290 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:56:59:309.324 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:56:59:309.339 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:56:59:309.351 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:56:59:309.363 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:56:59:312.821 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:56:59:312.928 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:56:59:312.968 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:56:59:313.005 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:56:59:313.054 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:56:59:313.299 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:56:59:313.324 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:56:59:313.337 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:56:59:313.403 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:56:59:313.417 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:56:59:441.701 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:56:59:441.863 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:56:59:441.905 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:56:59:441.945 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:56:59:441.970 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:56:59:442.225 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:56:59:442.276 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:56:59:442.407 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:56:59:442.422 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:56:59:442.448 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:57:04:309.247 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:57:04:309.450 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:57:04:309.479 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:57:04:309.526 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:57:04:309.568 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:57:04:309.596 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:57:04:309.629 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:57:04:309.645 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:57:04:309.658 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:57:04:309.670 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:57:04:313.362 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:57:04:313.461 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:57:04:313.501 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:57:04:313.531 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:57:04:313.571 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:57:04:313.731 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:57:04:313.760 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:57:04:313.827 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:57:04:313.883 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:57:04:313.935 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:57:04:442.301 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:57:04:442.464 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:57:04:442.522 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:57:04:442.557 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:57:04:442.587 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:57:04:442.756 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:57:04:442.788 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:57:04:442.802 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:57:04:442.816 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:57:04:442.831 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:57:06:060.249 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-17 Mon 01:57:06:060.522 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Connect : Start Connect timer 10.0.0.22:8086 2025-02-17 Mon 01:57:06:060.636 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:57:06:060.665 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: SANDESH: Send FAILED: 1739757426060357 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:57:06:060.683 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:57:06:060.692 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:57:06:060.743 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-17 Mon 01:57:06:060.772 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: -(-1):EvTcpConnectFail 2025-02-17 Mon 01:57:06:060.854 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: SANDESH: Send FAILED: 1739757426060621 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 46, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:57:06:060.873 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:57:06:060.883 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-17 Mon 01:57:06:060.894 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Connect : EvTcpConnectFail 2025-02-17 Mon 01:57:06:061.002 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-17 Mon 01:57:06:061.038 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Idle 2025-02-17 Mon 01:57:06:061.095 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:06:061.235 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: SANDESH: Send FAILED: 1739757426060959 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 48, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:57:06:061.249 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:57:06:061.260 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:06:061.280 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: SANDESH: Send FAILED: 1739757426060980 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:57:06:061.291 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:57:06:061.300 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 01:57:06:061.318 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:06:061.488 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: SANDESH: Send FAILED: 1739757426061087 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 48, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:57:06:061.506 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700113172224, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:57:09:218.533 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:09:218.653 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: SANDESH: Send FAILED: 1739757429218338 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 76 dequeues = 76 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 102 dequeues = 102 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739757426061037 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739757426061316 ] max_sm_queue_count = 5 ] 2025-02-17 Mon 01:57:09:218.682 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:57:09:218.692 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:09:218.837 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: SANDESH: Send FAILED: 1739757429218483 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 51, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:57:09:218.854 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:57:09:309.536 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:57:09:309.728 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:57:09:309.758 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:57:09:309.793 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:57:09:309.823 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:57:09:309.851 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:57:09:309.886 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:57:09:309.901 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:57:09:309.913 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:57:09:309.938 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:57:09:313.780 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:57:09:313.872 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:57:09:313.913 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:57:09:313.957 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:57:09:313.987 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:57:09:314.332 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:57:09:314.355 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:57:09:314.370 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:57:09:314.387 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:57:09:314.408 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:57:09:442.842 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:57:09:443.021 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:57:09:443.082 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:57:09:443.116 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:57:09:443.166 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:57:09:443.256 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:57:09:443.288 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:57:09:443.303 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:57:09:443.318 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:57:09:443.350 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:57:14:215.443 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700121569024, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:14:215.536 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700121569024, Pid 1]: SANDESH: Send FAILED: 1739757434215359 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.131848 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-02-17 Mon 01:57:14:215.548 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700121569024, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-02-17 Mon 01:57:14:309.823 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:57:14:310.110 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:57:14:310.155 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:57:14:310.213 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:57:14:310.259 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:57:14:310.299 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:57:14:310.346 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:57:14:310.372 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:57:14:310.392 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:57:14:310.411 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:57:14:314.395 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:57:14:314.487 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:57:14:314.547 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:57:14:314.603 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:57:14:314.644 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:57:14:314.965 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:57:14:315.271 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:57:14:315.308 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:57:14:315.425 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:57:14:315.453 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:57:14:443.384 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:57:14:443.551 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:57:14:443.592 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:57:14:443.622 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:57:14:443.651 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:57:14:443.792 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:57:14:443.822 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:57:14:443.877 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:57:14:443.892 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:57:14:443.905 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:57:19:310.209 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:57:19:310.419 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:57:19:310.450 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:57:19:310.485 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:57:19:310.522 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:57:19:310.538 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:57:19:310.548 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:57:19:310.575 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:57:19:310.605 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:57:19:310.630 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:57:19:315.021 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:57:19:315.306 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:57:19:315.356 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:57:19:315.394 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:57:19:315.453 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:57:19:315.499 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:57:19:315.618 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:57:19:315.659 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:57:19:315.774 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:57:19:315.788 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:57:19:443.870 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:57:19:444.036 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:57:19:444.078 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:57:19:444.112 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:57:19:444.146 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:57:19:444.297 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:57:19:444.346 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:57:19:444.365 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:57:19:444.379 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:57:19:444.405 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:57:24:310.515 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:57:24:310.771 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:57:24:310.807 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:57:24:310.847 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:57:24:310.888 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:57:24:310.918 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:57:24:310.968 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:57:24:310.984 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:57:24:310.997 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:57:24:311.008 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:57:24:315.432 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:57:24:315.641 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:57:24:315.689 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:57:24:315.800 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:57:24:315.840 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:57:24:315.858 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:57:24:315.948 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:57:24:315.965 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:57:24:316.125 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:57:24:316.226 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:57:24:444.406 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:57:24:444.567 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:57:24:444.614 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:57:24:444.642 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:57:24:444.672 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:57:24:444.773 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:57:24:444.808 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:57:24:444.829 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:57:24:444.845 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:57:24:444.858 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:57:29:310.855 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:57:29:311.108 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:57:29:311.141 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:57:29:311.180 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:57:29:311.210 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:57:29:311.238 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:57:29:311.272 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:57:29:311.287 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:57:29:311.299 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:57:29:311.311 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:57:29:315.885 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:57:29:315.992 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:57:29:316.028 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:57:29:316.148 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:57:29:316.246 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:57:29:316.370 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:57:29:316.396 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:57:29:316.421 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:57:29:316.511 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:57:29:316.571 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:57:29:444.846 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:57:29:445.021 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:57:29:445.062 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:57:29:445.096 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:57:29:445.125 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:57:29:445.253 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:57:29:445.299 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:57:29:445.314 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:57:29:445.327 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:57:29:445.339 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:57:34:311.189 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:57:34:311.411 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:57:34:311.446 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:57:34:311.495 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:57:34:311.528 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:57:34:311.558 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:57:34:311.593 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:57:34:311.609 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:57:34:311.621 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:57:34:311.633 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:57:34:316.421 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:57:34:316.524 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:57:34:316.560 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:57:34:316.592 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:57:34:316.620 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:57:34:316.866 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:57:34:316.913 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:57:34:316.940 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:57:34:317.092 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:57:34:317.106 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:57:34:445.345 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:57:34:445.478 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:57:34:445.525 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:57:34:445.554 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:57:34:445.579 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:57:34:445.748 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:57:34:445.790 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:57:34:445.804 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:57:34:445.818 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:57:34:445.832 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:57:36:061.142 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-17 Mon 01:57:36:061.386 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Connect : Start Connect timer 10.0.0.56:8086 2025-02-17 Mon 01:57:36:061.526 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:57:36:061.571 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: SANDESH: Send FAILED: 1739757456061231 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:57:36:061.584 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:57:36:061.599 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:57:36:061.833 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: SANDESH: Send FAILED: 1739757456061507 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 54, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:57:36:061.864 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699907389184, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:57:36:065.094 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-17 Mon 01:57:36:065.128 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: -(-1):EvTcpConnectFail 2025-02-17 Mon 01:57:36:066.956 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-17 Mon 01:57:36:066.990 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Connect : EvTcpConnectFail 2025-02-17 Mon 01:57:36:067.184 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-17 Mon 01:57:36:067.227 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Idle 2025-02-17 Mon 01:57:36:067.322 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:36:067.579 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: SANDESH: Send FAILED: 1739757456067111 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 56, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:57:36:067.608 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:57:36:067.623 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:36:067.656 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: SANDESH: Send FAILED: 1739757456067152 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:57:36:067.666 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:57:36:067.679 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 01:57:36:067.713 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:36:067.954 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: SANDESH: Send FAILED: 1739757456067307 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 56, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:57:36:067.979 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:57:39:218.776 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700121569024, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-17 Mon 01:57:39:218.845 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:39:219.141 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: SANDESH: Send FAILED: 1739757459218628 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 84 dequeues = 84 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 113 dequeues = 113 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739757456067225 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739757456067711 ] max_sm_queue_count = 5 ] 2025-02-17 Mon 01:57:39:219.163 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:57:39:219.175 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:57:39:219.347 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: SANDESH: Send FAILED: 1739757459218755 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 59, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:57:39:219.377 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:57:39:311.493 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:57:39:311.683 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:57:39:311.710 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:57:39:311.744 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:57:39:311.774 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:57:39:311.803 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:57:39:311.836 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:57:39:311.852 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:57:39:311.864 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:57:39:311.888 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:57:39:316.904 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:57:39:317.009 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:57:39:317.039 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:57:39:317.117 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:57:39:317.151 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:57:39:317.293 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:57:39:317.332 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:57:39:317.347 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:57:39:317.433 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:57:39:317.448 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:57:39:445.851 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:57:39:446.039 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:57:39:446.093 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:57:39:446.135 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:57:39:446.164 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:57:39:446.451 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:57:39:446.500 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:57:39:446.518 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:57:39:446.534 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:57:39:446.553 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:57:44:311.754 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:57:44:311.964 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:57:44:311.995 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:57:44:312.033 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:57:44:312.065 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:57:44:312.105 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:57:44:312.140 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:57:44:312.155 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:57:44:312.168 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:57:44:312.180 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:57:44:317.350 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:57:44:317.459 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:57:44:317.495 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:57:44:317.523 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:57:44:317.554 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:57:44:317.751 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:57:44:317.783 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:57:44:317.810 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:57:44:317.825 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:57:44:317.898 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:57:44:446.534 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:57:44:446.659 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:57:44:446.694 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:57:44:446.722 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:57:44:446.749 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:57:44:446.881 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:57:44:446.907 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:57:44:446.921 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:57:44:446.948 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:57:44:446.961 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:57:49:312.041 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:57:49:312.230 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:57:49:312.258 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:57:49:312.293 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:57:49:312.349 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:57:49:312.381 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:57:49:312.415 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:57:49:312.431 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:57:49:312.444 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:57:49:312.455 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:57:49:317.805 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:57:49:317.892 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:57:49:317.942 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:57:49:317.971 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:57:49:318.001 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:57:49:318.348 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:57:49:318.373 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:57:49:318.388 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:57:49:318.442 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:57:49:318.541 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:57:49:446.975 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:57:49:447.120 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:57:49:447.160 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:57:49:447.189 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:57:49:447.216 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:57:49:447.404 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:57:49:447.443 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:57:49:447.457 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:57:49:447.470 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:57:49:447.482 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:57:54:312.332 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:57:54:312.553 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:57:54:312.585 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:57:54:312.629 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:57:54:312.662 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:57:54:312.691 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:57:54:312.727 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:57:54:312.745 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:57:54:312.758 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:57:54:312.770 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:57:54:318.411 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:57:54:318.516 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:57:54:318.552 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:57:54:318.579 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:57:54:318.614 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:57:54:318.835 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:57:54:318.870 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:57:54:318.886 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:57:54:318.901 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:57:54:318.949 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:57:54:447.508 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:57:54:447.658 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:57:54:447.721 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:57:54:447.774 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:57:54:447.814 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:57:54:447.951 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:57:54:447.988 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:57:54:448.009 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:57:54:448.066 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:57:54:448.083 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:57:59:312.638 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:57:59:312.811 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:57:59:312.837 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:57:59:312.876 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:57:59:312.916 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:57:59:312.973 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:57:59:313.024 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:57:59:313.044 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:57:59:313.062 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:57:59:313.079 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:57:59:318.892 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:57:59:318.996 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:57:59:319.045 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:57:59:319.087 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:57:59:319.128 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:57:59:319.289 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:57:59:319.326 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:57:59:319.375 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:57:59:319.411 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:57:59:319.465 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:57:59:448.050 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:57:59:448.201 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:57:59:448.243 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:57:59:448.273 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:57:59:448.300 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:57:59:448.485 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:57:59:448.515 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:57:59:448.529 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:57:59:448.542 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:57:59:448.629 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:58:04:312.887 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:58:04:313.052 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2d270 2025-02-17 Mon 01:58:04:313.083 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:58:04:313.123 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:58:04:313.155 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:58:04:313.193 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2c930 2025-02-17 Mon 01:58:04:313.208 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2bff0 2025-02-17 Mon 01:58:04:313.220 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2b6b0 2025-02-17 Mon 01:58:04:313.241 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:58:04:313.272 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ad70 2025-02-17 Mon 01:58:04:319.330 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:58:04:319.413 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:58:04:319.449 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:58:04:319.493 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:58:04:319.522 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:58:04:319.722 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29400 2025-02-17 Mon 01:58:04:319.751 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2a470 2025-02-17 Mon 01:58:04:319.862 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c29b50 2025-02-17 Mon 01:58:04:319.881 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c24da0 2025-02-17 Mon 01:58:04:319.909 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c28830 2025-02-17 Mon 01:58:04:448.548 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:58:04:448.648 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:58:04:448.680 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:58:04:448.716 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:58:04:448.746 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:58:04:448.858 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c300b0 2025-02-17 Mon 01:58:04:448.891 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2f770 2025-02-17 Mon 01:58:04:448.912 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2ee30 2025-02-17 Mon 01:58:04:448.937 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2e4f0 2025-02-17 Mon 01:58:04:448.951 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: -1 0x2c2dbb0 2025-02-17 Mon 01:58:06:068.869 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-17 Mon 01:58:06:069.133 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: Connect : Start Connect timer 10.0.0.53:8086 2025-02-17 Mon 01:58:06:069.247 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:58:06:069.276 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: SANDESH: Send FAILED: 1739757486068961 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.53:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:58:06:069.294 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:58:06:069.303 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:58:06:069.449 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: SANDESH: Send FAILED: 1739757486069228 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.53:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 61, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:58:06:069.483 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:58:06:069.527 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-02-17 Mon 01:58:06:069.535 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: -(-1):EvTcpConnectFail 2025-02-17 Mon 01:58:06:070.055 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-02-17 Mon 01:58:06:070.066 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Connect : EvTcpConnectFail 2025-02-17 Mon 01:58:06:070.170 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-02-17 Mon 01:58:06:070.204 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Idle 2025-02-17 Mon 01:58:06:070.269 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:58:06:070.424 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: SANDESH: Send FAILED: 1739757486070125 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 63, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:58:06:070.450 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:58:06:070.462 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:58:06:070.482 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: SANDESH: Send FAILED: 1739757486070148 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 01:58:06:070.489 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:58:06:070.501 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 01:58:06:070.510 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:58:06:070.636 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: SANDESH: Send FAILED: 1739757486070259 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 63, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:58:06:070.648 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:58:09:219.117 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700121569024, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:58:09:219.230 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700121569024, Pid 1]: SANDESH: Send FAILED: 1739757489218972 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 91 dequeues = 91 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 123 dequeues = 123 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739757486070203 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739757486070509 ] max_sm_queue_count = 5 ] 2025-02-17 Mon 01:58:09:219.254 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700121569024, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:58:09:219.264 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700121569024, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:58:09:219.410 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700121569024, Pid 1]: SANDESH: Send FAILED: 1739757489219078 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 66, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:58:09:219.424 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700121569024, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:58:09:313.132 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2d270 2025-02-17 Mon 01:58:09:313.344 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c2d270 2025-02-17 Mon 01:58:09:313.371 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:313.394 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:313.404 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2c930 2025-02-17 Mon 01:58:09:313.449 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2bff0 2025-02-17 Mon 01:58:09:313.493 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2b6b0 2025-02-17 Mon 01:58:09:313.531 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ad70 2025-02-17 Mon 01:58:09:313.606 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c2c930 2025-02-17 Mon 01:58:09:313.614 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:313.624 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:313.633 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c2bff0 2025-02-17 Mon 01:58:09:313.638 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:313.647 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:313.654 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c2b6b0 2025-02-17 Mon 01:58:09:313.659 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:313.667 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:313.674 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c2ad70 2025-02-17 Mon 01:58:09:313.692 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:313.700 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:313.734 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:313.744 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:313.813 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:313.830 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:313.857 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:313.924 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:313.992 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700164216576, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:58:09:314.056 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700164216576, Pid 1]: SANDESH: Send FAILED: 1739757489313889 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.56:6379::Down.10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 01:58:09:314.067 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700164216576, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:58:09:314.433 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:314.446 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:314.460 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:314.468 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:314.480 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:314.489 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:314.507 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:314.522 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:314.537 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:319.775 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29400 2025-02-17 Mon 01:58:09:319.883 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2a470 2025-02-17 Mon 01:58:09:319.963 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c29b50 2025-02-17 Mon 01:58:09:319.992 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c24da0 2025-02-17 Mon 01:58:09:320.020 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c28830 2025-02-17 Mon 01:58:09:320.183 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c29400 2025-02-17 Mon 01:58:09:320.191 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:320.210 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:320.336 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c2a470 2025-02-17 Mon 01:58:09:320.344 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:320.354 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:320.440 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c24da0 2025-02-17 Mon 01:58:09:320.447 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:320.455 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:320.536 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c29b50 2025-02-17 Mon 01:58:09:320.543 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:320.552 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:320.564 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c28830 2025-02-17 Mon 01:58:09:320.569 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:320.577 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:321.073 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:321.084 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:321.107 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:321.121 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:321.137 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:321.151 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:321.160 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:321.172 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:321.180 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:321.245 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:321.274 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700121569024, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:58:09:321.452 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:321.500 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:321.499 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700121569024, Pid 1]: SANDESH: Send FAILED: 1739757489321211 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 01:58:09:321.523 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:321.525 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700121569024, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:58:09:321.538 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:321.565 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:448.937 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c300b0 2025-02-17 Mon 01:58:09:449.073 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2f770 2025-02-17 Mon 01:58:09:449.109 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2ee30 2025-02-17 Mon 01:58:09:449.138 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2e4f0 2025-02-17 Mon 01:58:09:449.163 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c2dbb0 2025-02-17 Mon 01:58:09:449.385 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c300b0 2025-02-17 Mon 01:58:09:449.393 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:449.420 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:449.443 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c2f770 2025-02-17 Mon 01:58:09:449.449 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:449.458 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:449.466 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c2ee30 2025-02-17 Mon 01:58:09:449.471 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:449.479 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:449.487 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c2e4f0 2025-02-17 Mon 01:58:09:449.492 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:449.499 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:449.627 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: RAC_Connect status: 0 0x2c2dbb0 2025-02-17 Mon 01:58:09:449.633 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: Connected to REDIS... 2025-02-17 Mon 01:58:09:449.642 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 01:58:09:449.917 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:449.942 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:449.966 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:449.988 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:449.997 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:450.070 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:450.121 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:58:09:450.174 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: SANDESH: Send FAILED: 1739757489450036 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 01:58:09:450.184 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700134164224, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:58:09:450.539 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:450.552 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:450.565 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:450.574 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:450.590 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:450.604 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:09:450.617 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 01:58:09:450.626 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 01:58:09:450.641 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 01:58:14:215.861 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700155819776, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 01:58:14:215.962 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700155819776, Pid 1]: SANDESH: Send FAILED: 1739757494215823 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.131848 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-02-17 Mon 01:58:14:215.975 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700155819776, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-02-17 Mon 01:58:36:070.744 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-02-17 Mon 01:58:36:071.091 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Connect : Start Connect timer 10.0.0.22:8086 2025-02-17 Mon 01:58:36:071.231 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:58:36:071.273 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: SANDESH: Send FAILED: 1739757516070915 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 01:58:36:071.285 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 01:58:36:071.297 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 01:58:36:071.514 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: SANDESH: Send FAILED: 1739757516071212 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 72, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:58:36:071.550 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699303409408, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 01:58:36:074.227 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.53:47926::10.0.0.22:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-02-17 Mon 01:58:36:074.253 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: 10.0.0.53:47926::10.0.0.22:8086(-1) OnSessionEvent TCP Connected 2025-02-17 Mon 01:58:36:074.261 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700316285056, Pid 1]: 10.0.0.53:47926::10.0.0.22:8086(-1):EvTcpConnected 2025-02-17 Mon 01:58:36:074.508 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-17 Mon 01:58:36:074.541 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-17 Mon 01:58:36:074.578 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: ClientInit 2025-02-17 Mon 01:58:36:074.597 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2738-2.:contrail-query-engine:0:Database count 5 2025-02-17 Mon 01:58:36:074.680 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:58:36:074.690 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 01:58:36:074.713 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: SANDESH: Send FAILED: 1739757516074570 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 01:58:36:074.721 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 01:58:36:074.738 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:58:36:074.745 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-17 Mon 01:58:36:074.755 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 01:58:36:074.762 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-17 Mon 01:58:36:074.898 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: SANDESH: Send FAILED: 1739757516074671 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 153, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 74, _iter59->first = sent _iter59->second = 25, ] ] ] 2025-02-17 Mon 01:58:36:074.912 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-17 Mon 01:58:36:095.584 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: OnMessage control in state: ClientInit 2025-02-17 Mon 01:58:36:095.670 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-17 Mon 01:58:36:095.680 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-17 Mon 01:58:36:095.707 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Received Ctrl Message with size 0 2025-02-17 Mon 01:58:36:095.981 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 155870 calls = 23 average_bytes = 6776 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 91 dequeues = 91 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 123 dequeues = 123 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1739757486070203 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1739757486070509 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 16 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 118279 calls = 14 average_bytes = 8448 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 153, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 74, _iter59->first = sent _iter59->second = 25, ] ] max_sm_queue_count = 5 proxy seq 54 2025-02-17 Mon 01:58:36:096.012 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-17 Mon 01:58:36:096.038 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 37 2025-02-17 Mon 01:58:36:096.051 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-17 Mon 01:58:36:096.084 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2738-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.131848 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 5 2025-02-17 Mon 01:58:36:096.093 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-02-17 Mon 01:58:36:096.100 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-17 Mon 01:58:36:096.134 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: Established 2025-02-17 Mon 01:58:39:219.914 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-17 Mon 01:58:42:669.360 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: OnMessage control in state: Established 2025-02-17 Mon 01:58:42:669.432 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2025-02-17 Mon 01:58:42:669.442 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Established : EvSandeshMessageRecv 2025-02-17 Mon 01:58:42:669.488 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: Received Ctrl Message with size 3 2025-02-17 Mon 01:58:42:669.502 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 57, total = 0 2025-02-17 Mon 01:58:42:669.513 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 39, total = 0 2025-02-17 Mon 01:58:42:669.523 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 5, total = 0 2025-02-17 Mon 01:58:42:669.539 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-17 Mon 01:59:39:220.634 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-17 Mon 02:00:39:221.383 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700160018176, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-17 Mon 02:01:39:223.094 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700138362624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-17 Mon 02:02:39:223.859 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140699903190784, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-17 Mon 02:03:39:228.340 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700129965824, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-02-17 Mon 02:04:02:621.470 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: Processing scm::EvStop in state Established 2025-02-17 Mon 02:04:02:621.580 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: Idle 2025-02-17 Mon 02:04:02:621.692 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-02-17 Mon 02:04:02:621.704 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-02-17 Mon 02:04:02:621.862 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: SANDESH: Send FAILED: 1739757842621675 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1739757218995095 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 3463 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 476917 calls = 54 average_bytes = 8831 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 34 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 153, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 76, _iter59->first = sent _iter59->second = 59, ] ] ] 2025-02-17 Mon 02:04:02:621.888 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 140700172613376, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 02:04:32:297.422 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: http-server-port 8091 2025-02-17 Mon 02:04:32:298.675 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Max-tasks 16 2025-02-17 Mon 02:04:32:298.693 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Max-slice 100 2025-02-17 Mon 02:04:32:298.700 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Collectors 10.0.0.56:8086 2025-02-17 Mon 02:04:32:298.705 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Collectors 10.0.0.53:8086 2025-02-17 Mon 02:04:32:298.710 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Collectors 10.0.0.22:8086 2025-02-17 Mon 02:04:32:298.736 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: SANDESH: ROLE : Generator 2025-02-17 Mon 02:04:32:298.742 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-02-17 Mon 02:04:32:298.747 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2738-2. 2025-02-17 Mon 02:04:32:298.752 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: SANDESH: NODE TYPE : Database 2025-02-17 Mon 02:04:32:298.757 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: SANDESH: INSTANCE ID : 0 2025-02-17 Mon 02:04:32:298.763 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-02-17 Mon 02:04:32:298.769 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-02-17 Mon 02:04:32:299.523 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: HTTP Introspect Init 2025-02-17 Mon 02:04:32:299.648 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-02-17 Mon 02:04:32:299.733 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Sandesh Http Server Port: 8091 2025-02-17 Mon 02:04:32:299.787 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-02-17 Mon 02:04:32:299.801 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-02-17 Mon 02:04:32:299.906 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Idle 2025-02-17 Mon 02:04:32:300.105 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: SANDESH: No Client: 1739757872300025 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739757872299920 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 02:04:32:300.259 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-02-17 Mon 02:04:32:300.274 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-02-17 Mon 02:04:32:300.287 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Cassandra Servers: 10.0.0.56:9042 10.0.0.53:9042 10.0.0.22:9042 2025-02-17 Mon 02:04:32:300.505 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: Processing scm::EvStart in state Idle 2025-02-17 Mon 02:04:32:300.580 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: Disconnect 2025-02-17 Mon 02:04:32:300.649 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-02-17 Mon 02:04:32:300.853 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: Connect : Start Connect timer 10.0.0.56:8086 2025-02-17 Mon 02:04:32:300.922 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 02:04:32:300.942 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: SANDESH: Send FAILED: 1739757872300572 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-17 Mon 02:04:32:300.951 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 02:04:32:300.959 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 02:04:32:301.479 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: SANDESH: Send FAILED: 1739757872300639 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739757872299920 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 02:04:32:301.507 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 02:04:32:301.520 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 02:04:32:301.601 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: SANDESH: Send FAILED: 1739757872300696 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739757872299920 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 02:04:32:301.615 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 02:04:32:301.624 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 02:04:32:301.639 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: SANDESH: Send FAILED: 1739757872300714 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-02-17 Mon 02:04:32:301.646 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 02:04:32:301.654 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 02:04:32:301.717 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: SANDESH: Send FAILED: 1739757872300913 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1739757872299920 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 02:04:32:301.735 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-02-17 Mon 02:04:32:413.560 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-02-17 Mon 02:04:32:417.916 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-02-17 Mon 02:04:32:417.986 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: SANDESH: Send FAILED: 1739757872417827 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-17 Mon 02:04:32:417.999 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-02-17 Mon 02:04:32:419.868 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.53:35498::10.0.0.56:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2025-02-17 Mon 02:04:32:419.891 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: 10.0.0.53:35498::10.0.0.56:8086(-1) OnSessionEvent TCP Connected 2025-02-17 Mon 02:04:32:419.908 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: 10.0.0.53:35498::10.0.0.56:8086(-1):EvTcpConnected 2025-02-17 Mon 02:04:32:419.947 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: RAC_Connect status: 0 0x2552c70 2025-02-17 Mon 02:04:32:419.953 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:04:32:419.965 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:04:32:419.988 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: RAC_Connect status: 0 0x25534d0 2025-02-17 Mon 02:04:32:419.993 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:04:32:420.001 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:04:32:420.009 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: RAC_Connect status: 0 0x2553d50 2025-02-17 Mon 02:04:32:420.015 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:04:32:420.022 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:04:32:420.030 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: RAC_Connect status: 0 0x25545d0 2025-02-17 Mon 02:04:32:420.035 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:04:32:420.042 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:04:32:420.050 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: RAC_Connect status: 0 0x2554e50 2025-02-17 Mon 02:04:32:420.055 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:04:32:420.062 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.53 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:04:32:420.071 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: RAC_Connect status: 0 0x2515240 2025-02-17 Mon 02:04:32:420.076 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:04:32:420.084 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:04:32:420.092 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: RAC_Connect status: 0 0x2515a00 2025-02-17 Mon 02:04:32:420.098 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:04:32:420.105 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:04:32:420.113 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: RAC_Connect status: 0 0x2516210 2025-02-17 Mon 02:04:32:420.118 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:04:32:420.132 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:04:32:420.144 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: RAC_Connect status: 0 0x25556d0 2025-02-17 Mon 02:04:32:420.151 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:04:32:420.163 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:04:32:420.174 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: RAC_Connect status: 0 0x2516a90 2025-02-17 Mon 02:04:32:420.179 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:04:32:420.187 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:04:32:420.195 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: RAC_Connect status: 0 0x2517310 2025-02-17 Mon 02:04:32:420.200 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:04:32:420.208 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.56 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:04:32:420.216 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: RAC_Connect status: 0 0x2555f50 2025-02-17 Mon 02:04:32:420.221 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:04:32:420.229 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:04:32:420.236 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: RAC_Connect status: 0 0x25567d0 2025-02-17 Mon 02:04:32:420.241 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:04:32:420.248 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:04:32:420.257 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: RAC_Connect status: 0 0x2557050 2025-02-17 Mon 02:04:32:420.261 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:04:32:420.269 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:04:32:420.277 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: RAC_Connect status: 0 0x25578d0 2025-02-17 Mon 02:04:32:420.282 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: Connected to REDIS... 2025-02-17 Mon 02:04:32:420.292 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-02-17 Mon 02:04:32:420.365 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-02-17 Mon 02:04:32:420.397 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-02-17 Mon 02:04:32:420.433 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: ClientInit 2025-02-17 Mon 02:04:32:420.449 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2738-2.:contrail-query-engine:0:Database count 1 2025-02-17 Mon 02:04:32:420.544 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 02:04:32:420.554 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 02:04:32:420.575 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: SANDESH: Send FAILED: 1739757872420425 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-02-17 Mon 02:04:32:420.580 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:04:32:420.600 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:04:32:420.615 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:04:32:420.624 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:04:32:420.663 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:04:32:420.689 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:04:32:420.703 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:04:32:420.712 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:04:32:420.725 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:04:32:420.733 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:04:32:420.745 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:04:32:420.759 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.53:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:04:32:420.776 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:04:32:420.791 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:04:32:420.806 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:04:32:420.584 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 02:04:32:420.846 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 02:04:32:420.854 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-02-17 Mon 02:04:32:420.877 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 02:04:32:420.884 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-02-17 Mon 02:04:32:421.009 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: SANDESH: Send FAILED: 1739757872420532 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739757872299920 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-02-17 Mon 02:04:32:421.025 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-02-17 Mon 02:04:32:421.037 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 02:04:32:421.043 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 02:04:32:421.064 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: SANDESH: Send FAILED: 1739757872420653 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-02-17 Mon 02:04:32:421.071 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 02:04:32:421.083 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 02:04:32:421.089 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 02:04:32:421.106 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: SANDESH: Send FAILED: 1739757872420683 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.56:6379::Down.10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 02:04:32:421.113 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 02:04:32:421.222 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:04:32:421.260 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:04:32:421.278 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:04:32:421.288 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:04:32:421.302 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:04:32:421.310 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:04:32:421.324 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:04:32:421.333 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:04:32:421.345 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:04:32:421.353 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:04:32:421.368 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:04:32:421.387 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:04:32:421.414 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:04:32:421.423 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:04:32:421.435 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:04:32:421.443 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:04:32:421.456 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:04:32:421.465 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:04:32:421.526 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:04:32:421.612 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:04:32:421.626 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 02:04:32:421.642 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 02:04:32:421.674 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: SANDESH: Send FAILED: 1739757872421578 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.56:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 02:04:32:421.682 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 02:04:32:421.646 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:04:32:421.761 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:04:32:421.785 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:04:32:421.809 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:04:32:421.840 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:04:32:421.854 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:04:32:421.869 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970413045504, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-02-17 Mon 02:04:32:421.871 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:04:32:421.920 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-02-17 Mon 02:04:32:421.929 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.56:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-02-17 Mon 02:04:32:421.980 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-02-17 Mon 02:04:32:421.895 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970413045504, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-02-17 Mon 02:04:32:422.037 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970413045504, Pid 1]: SANDESH: Send FAILED: 1739757872421829 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-02-17 Mon 02:04:32:422.045 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970413045504, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-02-17 Mon 02:04:32:425.134 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: OnMessage control in state: ClientInit 2025-02-17 Mon 02:04:32:425.193 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970400450304, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-02-17 Mon 02:04:32:425.221 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970400450304, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-02-17 Mon 02:04:32:425.253 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970400450304, Pid 1]: Received Ctrl Message with size 0 2025-02-17 Mon 02:04:32:425.381 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970400450304, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2738-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1739757872299920 collector_name = collector_ip = 10.0.0.56:8086 collector_list= [ [ (*_iter40) = 10.0.0.56:8086, (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.53:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2025-02-17 Mon 02:04:32:425.410 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970400450304, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-02-17 Mon 02:04:32:425.437 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970400450304, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2738-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.56:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.56:9042, (*_iter6) = 10.0.0.53:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.56:6379, (*_iter6) = 10.0.0.53:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2025-02-17 Mon 02:04:32:425.449 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970400450304, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-02-17 Mon 02:04:32:425.457 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970400450304, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-02-17 Mon 02:04:32:425.464 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970400450304, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-02-17 Mon 02:04:32:425.491 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970400450304, Pid 1]: Established 2025-02-17 Mon 02:05:32:300.836 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970387855104, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-02-17 Mon 02:06:32:302.189 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970429839104, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-02-17 Mon 02:07:32:303.029 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970387855104, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-02-17 Mon 02:08:32:303.674 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970387855104, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-02-17 Mon 02:09:32:304.423 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970510071552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-02-17 Mon 02:10:32:305.505 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970413045504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-02-17 Mon 02:11:32:306.515 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970413045504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-02-17 Mon 02:12:32:307.226 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970421442304, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-02-17 Mon 02:13:32:307.855 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970392053504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-02-17 Mon 02:14:32:308.609 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970417243904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-02-17 Mon 02:15:32:309.741 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970434037504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-02-17 Mon 02:16:32:310.664 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-02-17 Mon 02:17:32:311.419 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970392053504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-02-17 Mon 02:18:32:312.367 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970425640704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-02-17 Mon 02:19:32:313.018 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970387855104, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-02-17 Mon 02:20:32:313.886 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970505873152, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-02-17 Mon 02:20:32:764.583 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.53:8091::10.0.0.53:60736 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-17 Mon 02:20:47:741.575 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.53:8091::10.0.0.53:36988 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-17 Mon 02:21:32:314.488 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970413045504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-02-17 Mon 02:22:14:861.690 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.53:8091::10.0.0.56:55464 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-17 Mon 02:22:32:315.180 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970434037504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-02-17 Mon 02:22:38:871.985 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.56 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 773 2025-02-17 Mon 02:22:38:872.778 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID 0ee301e8-ecd6-11ef-a778-00000a000038 src/contrail-analytics/contrail-query-engine/query.cc 1166 2025-02-17 Mon 02:22:38:872.824 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QEQueryLog: 0ee301e8-ecd6-11ef-a778-00000a000038 :0:json query is: { end_time: 1739758958789591 , enqueue_time: 1739758958871338 , query_metadata: {"enqueue_time": 1739758958871211} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1739758939344830 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "7a9b2a95-d775-4174-901b-a3b61556288c", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-17 Mon 02:22:38:873.153 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 880 2025-02-17 Mon 02:22:38:873.176 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 931 2025-02-17 Mon 02:22:38:873.225 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for 0ee301e8-ecd6-11ef-a778-00000a000038 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 940 2025-02-17 Mon 02:22:38:873.348 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970505873152, Pid 1]: [SYS_INFO]: QEQueryLog: 0ee301e8-ecd6-11ef-a778-00000a000038 :0:json query is: { end_time: 1739758958789591 , enqueue_time: 1739758958871338 , query_metadata: {"enqueue_time": 1739758958871211} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1739758939344830 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "7a9b2a95-d775-4174-901b-a3b61556288c", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-17 Mon 02:22:38:922.836 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:922.883 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:922.896 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:922.904 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:922.912 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 UNION between tables of sizes 0 and 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:922.921 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:922.929 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:922.948 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:922.957 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:922.967 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:923.036 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970425640704, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:923.046 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970425640704, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:923.054 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970425640704, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:923.061 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970425640704, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:923.068 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970425640704, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 UNION between tables of sizes 0 and 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:923.107 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970425640704, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:923.118 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970425640704, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 UNION between tables of sizes 5 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:923.127 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970425640704, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:923.136 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970425640704, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 UNION between tables of sizes 5 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:923.144 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970425640704, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:923.402 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970425640704, Pid 1]: [SYS_INFO]: QEQueryLog: 0ee301e8-ecd6-11ef-a778-00000a000038 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 7a9b2a95-d775-4174-901b-a3b61556288c src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:923.549 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970505873152, Pid 1]: [SYS_INFO]: QEQueryLog: 0ee301e8-ecd6-11ef-a778-00000a000038 ObjectVMTable :2:Column ObjectTypeName1 Table ObjectVMTable object_id 7a9b2a95-d775-4174-901b-a3b61556288c src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:923.566 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970425640704, Pid 1]: [SYS_INFO]: QEQueryLog: 0ee301e8-ecd6-11ef-a778-00000a000038 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 7a9b2a95-d775-4174-901b-a3b61556288c src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:923.782 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970425640704, Pid 1]: [SYS_INFO]: QEQueryLog: 0ee301e8-ecd6-11ef-a778-00000a000038 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 7a9b2a95-d775-4174-901b-a3b61556288c src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:924.014 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970425640704, Pid 1]: [SYS_INFO]: QEQueryLog: 0ee301e8-ecd6-11ef-a778-00000a000038 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 7a9b2a95-d775-4174-901b-a3b61556288c src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:924.163 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970425640704, Pid 1]: [SYS_INFO]: QEQueryLog: 0ee301e8-ecd6-11ef-a778-00000a000038 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 7a9b2a95-d775-4174-901b-a3b61556288c src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:38:923.580 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:924.304 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:924.323 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:924.336 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:924.347 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:924.358 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:924.369 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:924.380 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:924.391 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:38:924.401 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 0ee301e8-ecd6-11ef-a778-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:38:924.460 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QEQueryLog: 0ee301e8-ecd6-11ef-a778-00000a000038 :0:json query is: { end_time: 1739758958789591 , enqueue_time: 1739758958871338 , query_metadata: {"enqueue_time": 1739758958871211} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1739758939344830 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "7a9b2a95-d775-4174-901b-a3b61556288c", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-17 Mon 02:22:38:924.653 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 6 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 520 2025-02-17 Mon 02:22:38:924.754 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 6 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 528 2025-02-17 Mon 02:22:38:924.803 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":6} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 563 2025-02-17 Mon 02:22:38:925.295 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: [SYS_INFO]: QELog: Finished: QID 0ee301e8-ecd6-11ef-a778-00000a000038 Table ObjectVMTable Time(ms) 52 RedisTime(ms) 0 MergeTime(ms) 0 Rows 6 EnQ-delay1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 644 2025-02-17 Mon 02:22:38:925.333 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970510071552, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-2738-2. table = ObjectVMTable query_stats= [ qid = 0ee301e8-ecd6-11ef-a778-00000a000038 where = [[{"name": "ObjectId", "value": "7a9b2a95-d775-4174-901b-a3b61556288c", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 50,0, 49,0, 49,0, chunk_select_time = 0,0, 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 52 rows = 6 enq_delay = 1 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 636 2025-02-17 Mon 02:22:38:925.343 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 676 2025-02-17 Mon 02:22:41:958.231 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.56 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 773 2025-02-17 Mon 02:22:41:959.102 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID 10b9c7a6-ecd6-11ef-8423-00000a000038 src/contrail-analytics/contrail-query-engine/query.cc 1166 2025-02-17 Mon 02:22:41:959.143 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QEQueryLog: 10b9c7a6-ecd6-11ef-8423-00000a000038 :0:json query is: { end_time: 1739758961940746 , enqueue_time: 1739758961957349 , query_metadata: {"enqueue_time": 1739758961957092} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1739758939344830 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338:ctest-vn22-94765338", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-17 Mon 02:22:41:959.307 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 880 2025-02-17 Mon 02:22:41:959.323 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 931 2025-02-17 Mon 02:22:41:959.370 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for 10b9c7a6-ecd6-11ef-8423-00000a000038 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 940 2025-02-17 Mon 02:22:41:959.488 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970417243904, Pid 1]: [SYS_INFO]: QEQueryLog: 10b9c7a6-ecd6-11ef-8423-00000a000038 :0:json query is: { end_time: 1739758961940746 , enqueue_time: 1739758961957349 , query_metadata: {"enqueue_time": 1739758961957092} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1739758939344830 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338:ctest-vn22-94765338", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-17 Mon 02:22:42:011.655 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:42:011.700 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:42:011.709 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:42:011.716 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:42:011.740 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:42:011.751 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:42:011.761 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:42:011.771 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:42:011.782 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:42:011.790 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:42:013.784 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970413045504, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 UNION between tables of sizes 0 and 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:42:013.816 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970413045504, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:42:013.833 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970413045504, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:42:013.842 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970413045504, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:42:013.850 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970413045504, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:42:013.858 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970413045504, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:42:013.866 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970413045504, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:42:013.873 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970413045504, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:42:013.881 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970413045504, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:42:013.888 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970413045504, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:42:014.152 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970413045504, Pid 1]: [SYS_INFO]: QEQueryLog: 10b9c7a6-ecd6-11ef-8423-00000a000038 ObjectRoutingInstance :1:Column ObjectTypeName1 Table ObjectRoutingInstance object_id default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338:ctest-vn22-94765338 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-02-17 Mon 02:22:42:014.771 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:42:014.803 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:42:014.812 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:42:014.821 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:42:014.833 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:42:014.845 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:42:014.856 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:42:014.867 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:42:014.878 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-02-17 Mon 02:22:42:014.887 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: 10b9c7a6-ecd6-11ef-8423-00000a000038 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-02-17 Mon 02:22:42:014.934 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970518468352, Pid 1]: [SYS_INFO]: QEQueryLog: 10b9c7a6-ecd6-11ef-8423-00000a000038 :0:json query is: { end_time: 1739758961940746 , enqueue_time: 1739758961957349 , query_metadata: {"enqueue_time": 1739758961957092} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1739758939344830 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338:ctest-vn22-94765338", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-02-17 Mon 02:22:42:015.151 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 520 2025-02-17 Mon 02:22:42:015.191 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 528 2025-02-17 Mon 02:22:42:015.290 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":1} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 563 2025-02-17 Mon 02:22:42:016.742 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970396251904, Pid 1]: [SYS_INFO]: QELog: Finished: QID 10b9c7a6-ecd6-11ef-8423-00000a000038 Table ObjectRoutingInstance Time(ms) 57 RedisTime(ms) 0 MergeTime(ms) 0 Rows 1 EnQ-delay1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 644 2025-02-17 Mon 02:22:42:016.789 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970417243904, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-2738-2. table = ObjectRoutingInstance query_stats= [ qid = 10b9c7a6-ecd6-11ef-8423-00000a000038 where = [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-49880253:ctest-vn22-94765338:ctest-vn22-94765338", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 55,0, 54,0, 51,0, chunk_select_time = 0,0, 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 57 rows = 1 enq_delay = 1 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 636 2025-02-17 Mon 02:22:42:016.815 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970396251904, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 676 2025-02-17 Mon 02:23:32:315.806 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970413045504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-02-17 Mon 02:24:32:316.815 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970421442304, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-02-17 Mon 02:25:32:317.486 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970413045504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-02-17 Mon 02:26:32:318.081 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970400450304, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-02-17 Mon 02:27:32:318.686 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970425640704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-02-17 Mon 02:28:32:319.536 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970425640704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-02-17 Mon 02:29:32:320.237 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970417243904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-02-17 Mon 02:29:39:925.486 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.53:8091::10.0.0.56:34846 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-02-17 Mon 02:30:32:321.119 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970514269952, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2025-02-17 Mon 02:31:32:321.750 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970429839104, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2025-02-17 Mon 02:31:34:402.074 UTC cn-jenkins-deploy-platform-ansible-os-2738-2 [Thread 139970666330240, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.53:8091::10.0.0.53:51812 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256